Patent Description:
With the recent advance of digital technologies, various types of electronic devices are being widely used such as mobile communication terminals, smart phones, tablet personal computers (PCs), notebooks, personal digital assistants (PDA), wearable devices, digital cameras, personal computers (PCs) or the like.

In recent years, researches and developments for not only a call function, a multimedia play function (e.g., music play and video play), electronic payment and financial transaction but also an augmented reality (AR) function in an electronic device, and a use thereof, are being increased. Augmented reality can be a technology for synthesizing virtual related information (e.g., a text, an image, etc.) to a real thing (e.g., a real environment) and showing. Unlike virtual reality (VR) aiming at only a virtual space and thing, augmented reality can provide a virtual related object on an object called a real environment, and provide a user with additional information difficult to be acquired by the real environment only.

<CIT> discloses a method, system and computer program for organizing and visualizing display objects within a virtual environment.

<CIT> discloses an electronic device and a method of controlling the electronic device. The electronic device may group items based on a drag input on a virtual bookshelf having the items.

However, an augmented reality function provided in a conventional electronic device cannot provide a wide user experience because following an interactive scheme in a <NUM>-dimensional (2D) environment in interacting with a real object and virtual object provided in augmented reality. For example, as a complex selection menu has to be used in selecting a plurality of objects and performing a function related to them, the complexity of a <NUM>-dimensional (3D) environment is increased and thus many handling procedures may be needed for object selection and function selection. Accordingly, a user's need for decreasing the complexity and requiring intuitiveness at object and function use under the augmented reality environment is increasing.

Various embodiments disclose an interacting method and apparatus for picking an object more intuitively and simply in augmented reality.

Various embodiments disclose an interacting method and apparatus for picking a plurality of objects more intuitively and simply in augmented reality.

Various embodiments disclose a method and apparatus for picking a plurality of objects more intuitively and simply and performing a function related to them in augmented reality.

Various embodiments disclose a method and apparatus for picking a plurality of objects more intuitively and simply and offering information related to them in augmented reality.

Technological solutions the present document seeks to achieve are not limited to the above-mentioned technological solutions, and other technological solutions not mentioned above would be able to be clearly understood by a person having ordinary skill in the art from the following statement.

The solution to the problem is according to the independent claims.

An electronic device of various embodiments can include a display, a camera module and at least one processor. The at least one processor can be configured to display one or more real objects and one or more virtual objects through the display, and receive a first input for picking any one object from among the objects, and then receive a second input related to the picked object, and in response to a third input being received for grouping the objects picked by repeating the first input and the second input, analyze information about the grouped objects, and in response to a fourth input related to the grouped objects being received, offer a function extracted on the basis of analyzing the information about the grouped objects.

An operating method of an electronic device of various embodiments can include displaying one or more real objects and one or more virtual objects through a display of the electronic device, and receiving a first input for picking any one object from among the objects, and then receiving a second input related to the picked object, and in response to a third input being received for grouping the objects picked by repeating the first input and the second input, analyzing information about the grouped objects, and in response to a fourth input related to the grouped objects being received, offering a function extracted on the basis of analyzing the information about the grouped objects.

An electronic device of various embodiments and a method for the same can adjust a posture of the electronic device on the basis of a user input received in a state of displaying a preview image of a first image acquired through a first camera, thereby acquiring a second image having a second composition corresponding to a first composition of the first image through a second camera having a characteristic different from a characteristic of the first camera.

An effect obtainable from the present disclosure is not limited to the above-mentioned effects, and other effects not mentioned would be able to be apparently understood from the following statement by a person having ordinary skill in the art to which the disclosure pertains.

Various embodiments of the present document are mentioned below with reference to the accompanying drawings.

<FIG> is a perspective view of an electronic device <NUM> performing various functions in an augmented reality environment according to various embodiments.

Referring to <FIG>, the electronic device <NUM> can be an eyeglass form which includes a frame <NUM> supporting at least one or more displays and/or lenses <NUM> (referred below to as displays) and disposed ahead of user's eyes, and temples <NUM> and <NUM> connected to both sides of the frame <NUM> and hanged on user's ears. For example, the temples <NUM> and <NUM> connected to the both sides of the frame <NUM> of the electronic device <NUM> can be implemented as a mount portion (e.g., a band) capable of fixing the frame to a user's head. According to an embodiment, the electronic device <NUM> can include one or more sensor modules <NUM> and <NUM> disposed in at least one or more of the temples <NUM> and <NUM> and/or the frame <NUM>. In accordance with various embodiments, the display <NUM> can be implemented as one or more display and/or lens <NUM> and <NUM> portions, and perform a display and/or lens function according to an operation mode. According to some embodiment, the displays and/or lenses <NUM> and <NUM> can be integrated into one device performing various functions or be implemented as two or more separable devices.

According to an embodiment, the electronic device <NUM> can offer at least one of a see-through function for offering augmented reality (AR) or a see-closed function for offering virtual reality (VR). The see-through function, for example, can be defined as a function of forwarding a real external thing(s) to user's eyes through the display <NUM> while offering the real external thing or a virtual object or thing to a user by using visual or various sensible means. The see-through function can offer additional information about a really seen thing, etc., and/or images to the user. In another embodiment, the wearable electronic device <NUM> can offer additional information related to the see-through function to the user by using a hologram, etc. without the display <NUM> as well.

The see-closed function can offer two screens formed by the display <NUM> ahead of user's eyes respectively, thereby allowing a user to watch contents (e.g., a movie, streaming, broadcasting, a game, etc.) offered through the two screens. This see-closed function can offer immersion to the user by using an independent screen. According to some embodiment, the electronic device <NUM> can be designed in the form of mounting a display device replacing the display <NUM> as well. For example, the electronic device <NUM> can include one or two display devices which are disposed ahead of user's both eyes, respectively.

According to various embodiments, the wearable electronic device <NUM> can offer a see-through mode by using the display <NUM> or a camera (e.g., the sensor modules <NUM> and <NUM>) installed in the frame <NUM>. The see-through mode can display image data acquired from the cameras <NUM> and <NUM> through two screens formed by the display <NUM>. Through this, a user can experience an external virtual environment and at the same time, confirm a peripheral environment through images of the camera <NUM> and <NUM> according to need.

According to various embodiments, the electronic device <NUM> can offer mixed reality (MR) (or hybrid reality). The mixed reality (MR) can include an environment in which virtual reality (VR) is combined to the real world and thus, a real physical object and a virtual object can interact with each other. The mixed reality (MR) can include a meaning of augmented reality (AR) of adding virtual information on the basis of reality and augmented virtuality (AV) of adding reality information to a virtual environment. The mixed reality can offer a smart environment of naturally connecting reality and virtuality, to allow a user to enjoy a wide experience. For example, the mixed reality can communicate with a virtual pet animal which is put on a user's palm, or build a virtual game environment within a real room to play a game. For another example, the mixed reality can build an environment of trying rearranging indoor furniture virtually, or being as if remote people gather and work together.

The above-described head-mounted wearable electronic device <NUM> can be called a head-mounted device (e.g., a head-mounted device (HMD) or a head-mounted theater (HMT)). According to some embodiment, the wearable electronic device <NUM> can be defined as an electronic device excluding the display <NUM> as well.

According to some embodiment, the wearable electronic device <NUM> can be designed to mount one or more components included in a display device such as a smart phone, on the frame <NUM>. By this, an attachable and detachable display device and a structure related to this can be omitted.

According to various embodiments, a wearable electronic device designed to mount a display device replacing the display <NUM> can be wiredly or wirelessly connected with an external electronic device, and output contents received from the external electronic device through an output device such as the display, or transmit a user input provided from an input device, to the external electronic device.

<FIG> is a block diagram of a head-mounted electronic device according to various embodiments.

Referring to <FIG>, the head-mounted device <NUM> (e.g., <NUM> of <FIG>) of various embodiments can include a processor <NUM>, a memory <NUM>, a communication module <NUM>, a sensor module <NUM>, an input device <NUM>, a power management module <NUM>, an eye tracking module (i.e., an eye tracker) <NUM>, a vibrator <NUM>, a focusing module (i.e., an adjustable optics) <NUM> (or a lens assembly) or a battery <NUM>. In the block diagram of <FIG>, other components (e.g., a display) are not illustrated for description convenience's sake. In another embodiment, some of the components illustrated in the block diagram can be included in a frame (e.g., <NUM> of <FIG>) or be included in a display (e.g., <NUM> of <FIG>).

The processor <NUM>, for example, can include a micro controller unit (MCU), and drive an operating system (OS) or an embedded software program, to control many hardware components connected to the processor <NUM>.

The memory <NUM> can store various data which are used by at least one component (e.g., the processor <NUM> or the sensor module <NUM>) of the head-mounted device <NUM>. The data, for example, can include input data or output data about software (e.g., program and a command related to this). The memory <NUM> can include a volatile memory or a non-volatile memory.

The communication module <NUM> can electrically connect a frame (e.g., <NUM> of <FIG>) and a display (e.g., <NUM> of <FIG>) by using wired communication and/or wireless communication, to perform data transmission and/or reception. According to an embodiment, the communication module <NUM> can include a universal serial bus (USB) module <NUM>, a wireless fidelity (WiFi) module <NUM>, a Bluetooth (BT) module <NUM>, a near field communication (NFC) module <NUM> or a global positioning system (GPS) module <NUM>, etc. According to various embodiments, at least some of the WiFi module <NUM>, the BT module <NUM>, the GPS module <NUM> or the NFC module <NUM> can be included in one integrated chip (IC) or IC package.

The sensor module <NUM> can meter a physical quantity and detect an activation state of the head-mounted device <NUM>, and convert metered or detected information into an electric signal. The sensor module <NUM>, for example, can include at least one of an acceleration sensor <NUM>, a gyro sensor <NUM>, a geomagnetic sensor <NUM>, a magnetic sensor <NUM>, a proximity sensor <NUM>, a gesture sensor <NUM>, a grip sensor <NUM>, a biometric sensor <NUM>, or a pressure sensor <NUM>. According to an embodiment, at least one sensor (e.g., the proximity sensor <NUM>, the grip sensor <NUM> or the pressure sensor <NUM>, etc.) of the sensor module <NUM> can be used to detect the wearing or non-wearing of the head-mounted device <NUM>. According to an embodiment, at least one sensor (e.g., the acceleration sensor <NUM>, the gyro sensor <NUM> or the geomagnetic sensor <NUM>) of the sensor module <NUM> can be used to detect the movement of a head on which the head-mounted device <NUM> is worn. In some embodiment, at least some of the sensor module <NUM> can be included in a display device such as a smart phone which is attachable to and detachable from the head-mounted device <NUM>.

According to some embodiment, at least one of infrared ray (IR) recognition, pressurization recognition and/or a variation of capacitance (or permittivity) which are associated with user's wearing can be detected to detect the wearing or non-wearing of a user. The gesture sensor <NUM> can detect a gesture of a user's hand or finger, as an input motion for the head-mounted device <NUM>. According to some embodiment, the sensor module <NUM> can recognize user's biometric information by using a biometric sensor (or a biometric recognition sensor) <NUM> such as an e-nose sensor, an electromyography sensor (EMG sensor), an electromyography sensor (EEG sensor), an electrocardiogram sensor (ECG sensor), an iris sensor, etc. According to various embodiments, the sensor module <NUM> can further include at least one control circuit for controlling at least one sensor belonging therein.

According to an embodiment, the pressure sensor <NUM> can be offered in the plural mutually different and, for example, can include a plurality of pressure sensors. In response to the head-mounted device <NUM> being mounted on a head, the plurality of pressure sensors are pressurized by a face. Accordingly to this, the processor <NUM> can acquire pressure values through the plurality of pressure sensors, and determine an arrangement state of the head-mounted device <NUM> related with the head on the basis of the detected pressure values, and control other components (e.g., the power management module <NUM>) on the basis of this.

The input device <NUM> can include a touch pad or a button. The touch pad, for example, can recognize a touch input in at least one scheme among a capacitive overlay scheme, a pressure sensitive scheme, an infrared beam scheme, or an ultrasonic scheme. According to various embodiments, the touch pad can further include a control circuit as well. The capacitive touch pad is possible for physical contact or proximal recognition. The touch pad can further include a tactile layer as well, and can offer a tactile response to a user. The button, for example, can include a physical button, an optical key or a keypad, etc..

The eye tracking module <NUM>, for example, can track user's eyes by using at least one scheme among an electro-oculography (EOG) sensor, a coil system, a dual Purkinje system, bright pupil systems or dark pupil systems. According to various embodiments, the eye tracking module <NUM> can further include a micro camera for eye tracking as well.

The vibrator <NUM> may include at least one vibrator that is disposed in the head-mounted device <NUM>. The vibrator <NUM> may activate a vibrator in various patterns on the basis of various types of triggers generated from the head-mounted device <NUM> in accordance with the control of the processor <NUM>.

The focusing module <NUM> can measure a distance between user's both eyes (i.e., an inter-pupil distance (IPD)) and adjust a distance of a lens and a display position of a display device (e.g., <NUM> of <FIG>), wherein a user can enjoy a movie suitable to its own eyesight.

The power management module <NUM> can manage power supplied to the head-mounted device <NUM>. According to an embodiment, the power management module <NUM>, for example, can be implemented as at least a portion of a power management integrated circuit (PMIC).

The battery <NUM> can supply power to at least one component of the head-mounted device <NUM>. According to an embodiment, the battery <NUM> can include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.

<FIG> is a block diagram of an electronic device within a network environment, performing various functions in an augmented reality environment according to various embodiments. The electronic device <NUM> can be the display <NUM> of <FIG> or <FIG>. According to various embodiments, the electronic device <NUM> can be the head-mounted device <NUM> or <NUM> of <FIG> or <FIG>.

The program 340may be stored in the memory <NUM> as software, and may include, for example, an operating system (OS) <NUM>, middleware <NUM>, or an application <NUM>.

According to various embodiments, an electronic device (e.g., the electronic device <NUM> of <FIG>, the electronic device <NUM> of <FIG> or the electronic device <NUM> of <FIG>) can include a display (e.g., the display <NUM> of <FIG> or the display device <NUM> of <FIG>), a camera module (e.g., the cameras <NUM> and <NUM> of <FIG> or the camera module <NUM> of <FIG>) and at least one processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>).

The at least one processor <NUM> or <NUM> can be configured to display one or more real objects and one or more virtual objects through the display <NUM> or <NUM>, and receive a first input for picking any one object from among the objects, and then receive a second input related to the picked object, and in response to a third input being received for grouping the objects picked by repeating the first input and the second input, analyze information about the grouped objects, and in response to a fourth input related to the grouped objects being received, offer a function extracted on the basis of analyzing the information about the grouped objects.

According to various embodiments, the second input can be received in a state in which the first input is maintained.

According to various embodiments, the second input can be received corresponding to a series of input motions.

According to various embodiments, the second input can be received corresponding to motions which comprise a first motion of moving to a specific region in a state in which the first input is maintained and a second motion carried out for the any one object in the specific region.

According to various embodiments, at least one of the first to fourth inputs can be received through the camera module.

According to various embodiments, the first input can be received corresponding to a first motion of two or more fingers of one hand of a user.

According to various embodiments, the second input can be received corresponding to a motion of moving the one hand of the user to a region specified by a third motion of the other hand of the user, and a second motion of a finger of the one hand within the specific region.

According to various embodiments, the processor <NUM> or <NUM> can cancel the picking of a corresponding object correspondingly to the second motion of the finger of the one hand outside the specific region.

According to various embodiments, in response to a fifth input which corresponds to a fourth motion of the other hand being received before the reception of the fourth input, the processor <NUM> or <NUM> can cancel the picking of the grouped objects.

According to various embodiments, in response to an editing mode being inputted after the reception of the fourth input, the processor <NUM> or <NUM> can release the grouping and set wherein at least one of the grouped objects is pickable.

<FIG> is a flowchart for explaining an example of an operation of an electronic device according to various embodiments.

According to various embodiments, a processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>) of the electronic device (e.g., the electronic device <NUM> of <FIG>, the electronic device <NUM> of <FIG> or the electronic device <NUM> of <FIG>) can, in operation <NUM>, receive a first input related to an object. The object, for example, can include a real object or a virtual object in an AR environment. The object can, regardless of its term, for example, include an app or an object embodying a function such as an icon displaying a function.

The first input, for example, can include a motion of such as making proximal the thumb and the index finger (or a plurality of fingers excepting the thumb) within a predetermined range of an object and picking the object. The first input, for example, can include a motion of drawing (checking) a line in the form of, for example, the V related to an object by using one or more fingers. The first input, for example, can include a motion of touching or tapping an object with a hand including fingers.

In operation <NUM>, the processor <NUM> or <NUM> can receive a second input related to the object, which is a target of the first input received in operation <NUM>, and a specific region. The second input, for example, can be an input associating the target object with the specific region. The second input, for example, can be comprised of a series of input motions which include an input of a motion of user's specifying a specific region, and a region of a motion associating the target object with the specific region. The specific region, for example, can be a region that a user specifies through a specified motion. The specific region, for example, can be a region which is set with a specified range to a specific position according to a user's voice command. The specific region, for example, can be visualized, and displayed, in a form such as a palm, a tray or a cart. The specific region, for example, can be provided instantly as a user executes a pickup mode by a voice, a motion or a menu in order to pick and group an object, or be provided as the user performs a motion of tapping, touching or checking the object under the pickup mode. The motion of associating the picked object and the specific region, for example, can maintain a motion of picking the object by using the thumb and the index finger (or the plurality of fingers excluding the thumb), and include a motion of moving to a specific region and putting down the picked object (e.g., a motion of spreading out fingers). The motion of associating the picked object and the specific region, for example, can include a motion of tapping, touching or checking the specific region subsequent to a motion of tapping, touching or checking the specific object. The motion of associating the picked object and the specific region, for example, can include a motion of tapping, touching or checking the object after providing the specific region.

Operations <NUM> and <NUM> can be repeatedly performed and accordingly to this, a plurality of objects can be picked and specified.

In operation <NUM>, the processor <NUM> or <NUM> can receive a third input related to specified objects. In accordance with the third input, the processor <NUM> or <NUM> can group the specified objects. The third input, for example, can include a motion of grabbing a fist or cupping a palm. The third input, for example, can include a voice command of instructing grouping or completion. The third input, for example, can include a motion of tapping a specific region once or more (e.g., double tap) or a motion of touching and holding for a predetermined time or longer (e.g., long press). Until when the third input is received in operation <NUM>, operations <NUM> and <NUM> can be repeatedly performed and a plurality of objects can be additionally picked and specified.

In operation <NUM>, the processor <NUM> or <NUM> can confirm information about the specified objects. In response to the specified object being a virtual object, the processor <NUM> or <NUM> can confirm information about the virtual object. In response to being a real object, for example, the processor <NUM> or <NUM> can transmit an image of the real object to a server (e.g., the server <NUM> of <FIG>) or an external electronic device (e.g., the electronic device <NUM> or <NUM> of <FIG>) and receive information about the corresponding image. Operation <NUM> can be omitted or be integrated into operation <NUM> described later.

In operation <NUM>, the processor <NUM> or <NUM> can receive a fourth input related to the specified objects. The fourth input, for example, can be a repetition of an inverse motion, or the same motion, of the third input. The fourth input, for example, can be a motion of unclenching a fist. The fourth input, for example, can include a motion of tapping a specific region at plural times (e.g., double tap) or a motion of touching and holding for a predetermined time or longer (e.g., long press) or uttering a confirming (i.e., viewing) command by a voice. The fourth input, for example, can include a motion such as a gesture of attaching an object such as a tray or cart representing a specific region to a specified region such as an edge of a bottom, wall or screen.

As receiving the fourth input, the processor <NUM> or <NUM> can, in operation <NUM>, confirm information of the grouped objects. The processor <NUM> or <NUM> can display an image, and/or information, of the grouped objects. Also, the processor <NUM> or <NUM> can analyze the information of the grouped objects and extract common information or analogized information from the information of the grouped objects.

In operation <NUM>, the processor <NUM> or <NUM> can allow the electronic device <NUM>, <NUM> or <NUM> to perform a specified function on the basis of the information of the grouped objects.

The processor <NUM> or <NUM> can extract a list of functions corresponding to the common information or analogized information among the information of the grouped objects. In response to the corresponding function being plural, the processor <NUM> or <NUM> can present the corresponding functions, for example, in order of function executed most for the corresponding objects in the electronic device <NUM>, <NUM> or <NUM> or in order of function executed generally most or in order of user setting. In response to one function being extracted, the processor <NUM> or <NUM> can perform the corresponding function instantly or according to user selection. In response to a plurality of functions being extracted, the processor <NUM> or <NUM> can display and present a list of the extracted functions in specified order for the sake of user selection, or recommend any one or more functions among the plurality of functions to allow the user to select.

An example of a function capable of being presented on the basis of the information of the grouped objects is given as follows.

For example, in response to the grouped objects being characters, a file share function such as a contact, a schedule or a photo taken together can be presented. For example, in response to a product or thing being grouped together with the character, when the product is an exercise tool, for example, an exercise method can be recommended and, when the product is clothes, shoes or a bag, a virtual fitting function can be recommended. For example, in response to a specific place being grouped together with the character, a function for setting a visit record to the specific place can be recommended. For example, in response to virtual objects such as an image, an app and a virtual thing being picked together with the character, a function of synthesizing the corresponding picked virtual object and the character, searching for an image of the corresponding character, executing a corresponding app function for the corresponding character, scheduling with a calendar and/or providing a synthesized image together with a virtual character can be recommended.

For example, in response to the product or thing being selected in the plural, a function of comparing the corresponding products or an electronic payment function can be recommended. For example, in response to virtual objects such as an image, an app and/or a button being grouped together with the product or thing, the product and the image or a movie can be synthesized to be utilized as an AR widget or a function offered by the button can be applied to the corresponding product to change an AR environment.

For example, in response to food being grouped together with food, a freezer, an oven and a place, a function for a cooking method, a safekeeping method, a visit place and a menu record can be recommended. For example, in response to a place and a place being grouped, a function for confirming a travel trajectory can be recommended. For example, in response to a plurality of devices being grouped, a function for controlling the devices can be recommended. For example, in response to the device being grouped together with a virtual object such as an image, an app and a button, a function of playing the corresponding image in the corresponding device, a control function such as device use reservation and time setting, a function of displaying the corresponding image as a virtual image in the corresponding device or various device control functions can be recommended.

For example, in response to an image being grouped together with other images, an app, a virtual object or buttons, a function of foldering, simultaneously playing several contents by a multi window, contents connecting, synthesizing or editing and a slide show function can be recommended. For example, in response to an app being grouped with a specific button, a function for changing a scale, location or color of a virtual object can be recommended. For example, in response to a place such as a room being grouped with a button, a function capable of controlling light or a temperature within the room can be recommended. For example, in response to a button and a button being grouped, an individualized multi-function control remote controller can be recommended.

On the other hand, in operation <NUM>, in response to a fifth input related to the specified objects being received before a fourth input related to the specified objects is received, the processor <NUM> or <NUM> can release the specialization of the objects. The processor <NUM> or <NUM> can delete temporarily stored object information about picked objects. The fifth input, for example, can be a motion of turning the spread-out palm upside down. As the motion of turning the palm upside down is received, the processor <NUM> or <NUM> can display an image in which a palm, tray or cart shaped object turns upside down and objects being picked and safely kept in a specific region fall down.

Though not illustrated, even after specified objects are grouped as the fourth input related to the specified objects is received, a user can pick by, for example, touching and holding for a predetermined time or longer (long press) a voice command for editing or addition or at least some of displayed objects or a specific region, thereby allowing to enter an editing mode, and can allow to eliminate the picked at least some objects from a group, or pick and add other objects to the group.

<FIG> are diagrams for explaining an example of an object pickup operation of an electronic device according to various embodiments.

Referring to <FIG>, a processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>) of the electronic device (e.g., the electronic device <NUM> of <FIG>, the electronic device <NUM> of <FIG> or the electronic device <NUM> of <FIG>), for example, can confirm a finger shape of a user's hand <NUM> and recognize a state of spreading out the thumb <NUM> and the index finger <NUM> as a waiting state for object picking, and can determine the user's thumb <NUM> and/or index finger <NUM> or an object located in a region of a predetermined range from these, as objects being in a pick position.

Referring to <FIG>, in response to the thumb <NUM> and index finger <NUM> of the user's hand <NUM> approaching within a predetermined distance range of an object <NUM> in a state in which they are spread out, the processor <NUM> or <NUM> can, for example, highlight the corresponding object <NUM> and wait for a pickup motion for picking the corresponding object <NUM>.

Referring to <FIG>, in response to the thumb <NUM> and index finger <NUM> of the user's hand <NUM> getting close within a predetermined range or more, the processor <NUM> or <NUM> can determine that an input for picking the corresponding object <NUM> which is on pickup standby is received.

<FIG> is a flowchart for explaining an example of an object pickup operation of an electronic device according to various embodiments.

In operation <NUM>, for example, in response to it being confirmed that a specific object is picked according as a first input is received in operation <NUM> of <FIG>, a processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>) of the electronic device (e.g., the electronic device <NUM> of <FIG>, the electronic device <NUM> of <FIG> or the electronic device <NUM> of <FIG>) confirms information of the picked object. In case of a virtual object, object information can include image information for rendering a 3D virtual object, and attribute and text information including a name, type and characteristic of an object. In case of a real object, object information may not exist in the electronic device <NUM>, <NUM> or <NUM>.

In operation <NUM>, the processor <NUM> or <NUM> can determine whether the picked object is a virtual object. In response to not being the virtual object, in operation <NUM>, the processor <NUM> or <NUM> can acquire an object image of a real object. For this, the processor <NUM> or <NUM> can capture a region of a predetermined range from an image acquired by a camera (e.g., the camera <NUM> or <NUM> of <FIG> or the camera module <NUM> of <FIG>), for example, from a position of a user's hand or finger. The processor <NUM> or <NUM> can calculate the position of the user's hand or finger in an image including objects acquired by the camera module <NUM>, and capture a region of a predetermined range centering on the calculated position and acquire the same as an object image. The processor <NUM> or <NUM> can extract a characteristic of an object from the acquired object image and check whether the extracted characteristic is consistent with an object characteristic stored in a memory (e.g., the memory <NUM> of <FIG> or the memory <NUM> of <FIG>).

On the other hand, in operation <NUM>, the processor <NUM> or <NUM> can transmit the acquired object image, for example, to a server (e.g., the server <NUM> of <FIG>) and, in operation <NUM>, can receive related information about the transmitted image from the server <NUM>. In response to the picked object being the virtual object, in operation <NUM>, the processor <NUM> or <NUM> can confirm information of the virtual object from the memory <NUM> or <NUM>.

As the second input is received in operation <NUM> of <FIG>, the processor <NUM> or <NUM> can determine that an object holding motion is inputted, and list and separately temporarily store the received or confirmed object related information.

<FIG> are diagrams for explaining an example of an object holding operation of an electronic device according to various embodiments.

Referring to <FIG>, as a motion of spreading out a palm of a user's hand <NUM> is sensed, the processor <NUM> or <NUM> can set a region of a predetermined range from the user's hand <NUM> as a specific region <NUM>. The processor <NUM> or <NUM> can vary the set specific region <NUM> in color or replace the set specific region <NUM> with a tray or cart shape and display. Referring to <FIG>, the set specific region <NUM> can be changed according to the direction of the user's hand <NUM>. In accordance with an embodiment, it can be implemented that while a motion of the user's hand <NUM> is maintained, an object can be additionally picked. In accordance with an embodiment, it can be implemented that even in response to the user's hand <NUM> motion not being maintained exactly, the set region <NUM> can be fixed and maintained, while the user's other hand <NUM> is hovering so as to pick an object or until before an object grouping command or an object pick completion command is received by the user's hand <NUM> or by another input.

Referring to <FIG>, the user's hand <NUM> for setting the specific region <NUM> and the user's hand <NUM> for picking the object <NUM> can be hands different from each other. A user can spread out the thumb <NUM> and the index finger <NUM> within a range of the specific region <NUM> with respect to the object <NUM> that the user picks with the thumb <NUM> and index finger <NUM> of the other hand <NUM> and moves to and draws into the specific region <NUM>, thereby associating the object <NUM> with the specific region <NUM>. For example, the processor <NUM> or <NUM> can display an image in which the thumb <NUM> and index finger <NUM> of the user's hand <NUM> are spread out and the corresponding object <NUM> is held within the specific region <NUM>, thereby showing that the corresponding object <NUM> is separately on standby for the sake of grouping within the specific region <NUM>.

<FIG> is a flowchart for explaining an example of an object holding operation of an electronic device according to various embodiments.

Referring to <FIG>, a processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>) of the electronic device (e.g., the electronic device <NUM> of <FIG>, the electronic device <NUM> of <FIG> or the electronic device <NUM> of <FIG>), for example, can confirm that a specific object is picked according as a first input is received in operation <NUM> of <FIG>. In operation <NUM>, the processor <NUM> or <NUM> can check whether the first input is maintained. Then, in operation <NUM>, the processor <NUM> or <NUM> can receive, for example, a second input of operation <NUM> of <FIG> in a state in which the first input is maintained. In operation <NUM>, the processor <NUM> or <NUM> can check whether a position where the second input received after the maintaining of the first input is provided is within a specific region. In response to the position where the second input is provided being within the specific region, in operation <NUM>, the processor <NUM> or <NUM> can add the picked object which is a grouping target, to a list. On the other hand, in response to the position where the second input is provided not being within the specific region, in operation <NUM>, the processor <NUM> or <NUM> can release the picking of the corresponding object. Accordingly, the user can provide a second input outside the specific region with respect to the object picked according to the first input, thereby simply canceling the picking of the corresponding object.

<FIG> are diagrams for explaining an example of an object grouping and confirming operation of an electronic device according to various embodiments.

Referring to <FIG>, for example, a user can maintain a motion (A) of spreading out a palm of a hand <NUM> recognized as a tray, thereby inputting a command of keeping possessing or holding picked objects in a specific region.

Referring to <FIG>, for example, the user can perform a motion (B) of changing the palm of the hand <NUM> recognized as the tray from the spread-out state (A) and cupping the palm or grabbing a fist, thereby performing possessing or holding in the specific region and inputting a grouping command for objects which are on grouping standby and accordingly to this, can complete the grouping of a plurality of objects.

Referring to <FIG>, for example, the user can change a state from the motion (B) of grabbing the fist of the hand <NUM> in <FIG> and perform a motion (C) of unclenching the fist of the hand <NUM> and spreading out the palm, thereby inputting a command of confirming (i.e., viewing) the grouped objects <NUM>, <NUM> and <NUM>.

<FIG> and <FIG> are diagrams for explaining an example of an object pick canceling operation of an electronic device according to various embodiments.

Referring to <FIG>, for example, a user can perform a motion (A) of turning a palm surface upside down from a state of spreading out a palm of a hand <NUM> recognized as a tray, thereby releasing all the picking of the picked object. On the other hand, for example, the user can maintain the motion (A) of turning the palm surface upside down for a predetermined time or longer in a state of a motion (e.g., a motion (C) of <FIG>) of spreading out the palm in order to input a command of confirming grouped objects in a state in which grouping is completed, thereby releasing the grouping and releasing all the picking of the picked object.

Referring to <FIG>, for example, the user can pick by performing a motion of being as if the user gets the thumb <NUM> and index finger <NUM> of the other hand <NUM> approach closely and picks an object <NUM> held in a tray region or a specific region <NUM> in a state of spreading out the palm of the hand <NUM> recognized as the tray, and move in this state and perform a motion of being as if the user spreads out the thumb <NUM> and the index finger <NUM> outside the specific region <NUM> and drops the corresponding object <NUM>, thereby allowing to selectively cancel the picking of some of grouping-target objects which are picked and held.

<FIG> is a flowchart for explaining an example of an object pick canceling or object pick adding operation of an electronic device according to various embodiments.

Referring to <FIG>, a processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>) of the electronic device (e.g., the electronic device <NUM> of <FIG>, the electronic device <NUM> of <FIG> or the electronic device <NUM> of <FIG>) can, even after a grouping command is inputted and grouping is completed, execute an editing mode, to cancel the picking of a picked object or additionally pick an object.

For example, after the fourth input related to the specified objects is received according to operation <NUM> of <FIG> and the grouping of the objects is completed, a motion of confirming (i.e., viewing) information about the grouped objects can be performed in operation <NUM> of <FIG>.

In operation <NUM>, the processor <NUM> or <NUM> can display the information about the grouped objects, to allow to confirm the grouped objects.

A user can confirm the information about the grouped objects, and input a voice command for editing or adding or touch and hold at least some of displayed (viewed) objects for a predetermined time or longer (long press) or touch and hold a tray region or specific region for a predetermined time or longer (long press), thereby entering an editing mode.

In response to it being determined that an editing mode execution command is inputted in operation <NUM>, the processor <NUM> or <NUM> proceeds to the editing mode and deletes a specific object from a grouping list or adds to the grouping list according to a user's input.

In operation <NUM>, the processor <NUM> or <NUM> can determine whether an editing mode ending or grouping command is received and complete the grouping.

In response to the grouping being completed, the processor <NUM> or <NUM> can, in operation <NUM>, perform a specified function on the basis of grouped object information. The processor <NUM> or <NUM> can present a recommended function, to allow a user to pick.

<FIG> is a flowchart for explaining an example of an operation of performing a function which is based on a grouped object.

In operation <NUM>, a processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>) can analyze information of grouped objects. Accordingly to this, the processor <NUM> or <NUM> can extract common information or analogized information from the information of the grouped objects.

In operation <NUM>, the processor <NUM> or <NUM> can check whether a specified function which will be executed by the electronic device (e.g., the electronic device <NUM> of <FIG>, the electronic device <NUM> of <FIG> or the electronic device <NUM> of <FIG>) exists on the basis of the information of the grouped objects.

In response to the specified function existing, the processor <NUM> or <NUM> can, in operation <NUM>, perform the specified function instantly or by user selection. On the other hand, in response to a function corresponding to the common information or analogized information among the information of the grouped objects existing in the plural, the processor <NUM> or <NUM> can display and present the corresponding functions in sequence in order of function executed most for the corresponding objects or in order of function executed generally most or according to user setting order, thereby allowing a user to select. In response to only one function existing, the processor <NUM> or <NUM> can allow to perform the corresponding function instantly or according to user selection.

On the other hand, in response to it being determined that the specified function does not exist on the basis of the analyzed information, the processor <NUM> or <NUM> can, in operation <NUM>, display a list of many executable functions and allow the user to specify a function the user intends to execute. In operation <NUM>, the processor <NUM> or <NUM> can check whether the user specifies the function the user intends to execute among the presented functions and, in response to the execution function being specified, the processor <NUM> or <NUM> can return to operation <NUM> and perform the specified function.

On the other hand, in response to the function executable for the grouped objects not existing or not being specified by the user, the processor <NUM> or <NUM> can, in operation <NUM>, provide a folder and store the corresponding grouped objects in the folder. Also, the processor <NUM> or <NUM> can display the corresponding folder as a non-executed folder or a suspended folder.

<FIG> are diagrams for explaining an example of picking and grouping a plurality of objects and performing a function which is based on the grouped objects in an electronic device according to various embodiments.

Referring to <FIG>, to pick a real object <NUM> among a virtual object or real object presented through a display (e.g., the display <NUM> of <FIG> or the display device <NUM> of <FIG>), a user can, for example, get a hand <NUM> approach within a predetermined range of the real object <NUM> and perform a pickup motion by using the thumb and the index finger, to execute a first input. The first input for the picking of the real object <NUM>, for example, can include a motion of drawing (checking), for example, in the form of the V for the object <NUM> by using one or more fingers, and/or a motion of touching or tapping the object <NUM> with a hand.

Referring to <FIG>, the user can perform a motion for a second input for associating the object <NUM> and a specific region <NUM> in a state of maintaining the first input for picking the object <NUM>. The second input for associating the object <NUM> and the specific region <NUM>, for example, can include a series of input motions of moving the right hand <NUM> and putting down the picked object <NUM> within the specific region <NUM> within a predetermined range from a palm specified by user's spreading out the palm of the left hand <NUM>, for example, within a region of the left hand <NUM>. The specific region, for example, can be visualized, and displayed, in a form and color such as a circle or oval and a tray or cart around the palm. The second input motion for associating the picked object <NUM> and the region of the left hand <NUM>, for example, can include two or more motions of tapping, touching or checking the region of the left hand <NUM> subsequent to a motion of tapping, touching or checking the corresponding object <NUM>.

Referring subsequently to <FIG>, the user can maintain a state of spreading out the left hand <NUM> and select an icon <NUM> of an electronic payment app among many applications by using the right hand <NUM> and move to the region of the left hand <NUM> and put down within the region, thereby additionally selecting the electronic payment app <NUM> as a grouping target. Milk <NUM> being the real object and the electronic payment app <NUM> being a virtual object can be selected and included in the region of the left hand <NUM>.

Referring to <FIG> and <FIG>, to group the picked objects <NUM> and <NUM> or complete the picking, for example, the user can perform a motion of cupping the left hand <NUM> and grabbing a fist and perform a motion for a third input. The third input, for example, can include a motion of tapping the region of the left hand <NUM> twice or more (e.g., double tap) or a motion of touching and holding for a predetermined time or longer (e.g., long press), in addition to a motion of grabbing the fist or cupping the palm.

The processor (e.g., the processor <NUM> of <FIG> or the processor <NUM> of <FIG>) can confirm and analyze information about the specified objects. In response to the specified object being a virtual object, the processor <NUM> or <NUM> can confirm information about the virtual object. In response to being a real object, for example, the processor <NUM> or <NUM> can transmit an image of the real object to a server (e.g., the server <NUM> of <FIG>) or an external electronic device (e.g., the electronic device <NUM> or <NUM> of <FIG>) and receive information about the corresponding image.

The processor <NUM> or <NUM> can extract a list of functions corresponding to common information or analogized information among the information of the grouped objects <NUM> and <NUM>. Through information analysis for the milk <NUM> and the electronic payment app <NUM>, the processor <NUM> or <NUM>, for example, can present a function for making electronic payment for, for example, milk <NUM>.

Referring to <FIG>, the user can perform, for example, a motion of spreading out the palm of the left hand <NUM> as a fourth input for confirming (viewing) the grouped objects <NUM> and <NUM>. Besides this, the fourth input, for example, can include a motion of tapping the region of the left hand <NUM> at plural times (e.g., double tap) or a motion of touching and holding for a predetermined time or longer (e.g., long press) or uttering a confirming (viewing) command by a voice. Also, the fourth input, for example, can include a motion such as a gesture of attaching an object such as a tray or cart representing a specific region, to a specified region such as an edge of a bottom, wall or screen.

In response to receiving the fourth input, the processor <NUM> or <NUM> can display images, and/or information, of the grouped objects, or the electronic device <NUM>, <NUM> or <NUM> can offer a specified function on the basis of the information of the grouped objects.

Referring to <FIG>, the processor <NUM> or <NUM> can offer a function <NUM> of making electronic payment for milk which can be analogized through language completion from the information (milk and electronic payment) of the grouped objects.

Referring to <FIG>, the processor <NUM> or <NUM> can display an object <NUM> representing a function of making payment for milk and, for example, the user can select a milk payment function <NUM> by using the hand <NUM>.

In response to user's selecting of the milk payment function <NUM>, referring to <FIG>, the processor <NUM> or <NUM> can execute the milk payment function and display (<NUM>) the execution result.

<FIG> are diagrams for explaining another example of picking and grouping a plurality of objects and performing a function which is based on the grouped objects in an electronic device according to various embodiments.

Referring to <FIG> and <FIG>, a user can pick virtual objects including a plurality of app icons and hold in a region of a left hand <NUM>, by using a right hand <NUM>. The picked plurality of virtual objects <NUM>, <NUM> and <NUM> can be displayed in the region of the left hand <NUM>.

Referring to <FIG>, the user can complete the object pickup and perform grouping, through a motion of cupping the left hand <NUM> and grabbing a fist. In this case, common information between the grouped objects is an app, and a function which can be presented may not exist according to the common information or analogized information. In this case, the processor <NUM> or <NUM> can provide a new folder <NUM> and move and store the grouped apps <NUM>, <NUM> and <NUM> in the folder <NUM>. Also, the processor <NUM> or <NUM> can allow to specify a name of the provided folder, or mark and display as a temporary folder.

<FIG> are diagrams for explaining a further example of picking and grouping a plurality of objects and performing a function which is based on the grouped objects in an electronic device according to various embodiments.

Referring to <FIG>, a user can perform a motion for picking specific characters among many characters and holding in a region of a left hand <NUM> by using a right hand <NUM>. Images <NUM>, <NUM> and <NUM> of the picked plurality of real objects can be displayed in the region of the left hand <NUM>.

Referring to <FIG>, the user can complete the object pickup and perform grouping, through a motion (A) of cupping the left hand <NUM> and grabbing a fist. In this case, common information between the grouped objects is a character, and a function which can be presented can be specified according to the common information or analogized information, or a function often used for the character can be presented. For example, in response to file share being specified or being often used for the character, referring to <FIG>, the processor <NUM> or <NUM> can recommend a file share app function <NUM>. In response to the user selecting the corresponding app <NUM> by using the right hand <NUM>, the processor <NUM> or <NUM> can activate the characters <NUM>, <NUM> and <NUM> and the file share app, and list and display (<NUM>) sharable files. In accordance with an embodiment, by an app targeting a character, a plurality of functions such as a video conference function and/or a collective talk function besides the file share app function can be extracted. In response to the plurality of functions being extracted, when there is not a previously specified function, for example, the functions can be presented to the user in order of function often used, to allow the user to select.

Claim 1:
An electronic device (<NUM>) for augmented reality comprising:
a display (<NUM>);
a camera module (<NUM>); and
at least one processor (<NUM>),
wherein the at least one processor (<NUM>) is configured to:
display objects through the display (<NUM>), wherein the objects include one or more real objects and one or more virtual objects;
receive a first input for picking any one object from the objects, and then
receive a second input for associating the picked object with a specific region;
receive a third input for grouping the objects picked by repeating the first input and the second input, wherein the grouped objects include a real object and a virtual object, wherein the third input is received corresponding to a motion of changing a palm of one hand of a user from a spread-out state and cupping the palm or grabbing a fist of the one hand of the user;
in response to the third input, analyze information of the grouped objects and extract information about the grouped objects, wherein the extracted information includes at least inferred information; and
in response to a fourth input related to the grouped objects being received, execute a function which is corresponding to the extracted information about the grouped objects, wherein the fourth input includes a confirming command by a voice, and
wherein the specific region is displayed in a form of a tray around the palm.