Audio providing method and device therefor

An electronic device may comprise: a sensing unit for acquiring image data and sensing an object and a user; an output unit for outputting audio; and a processor for receiving image data from the sensing unit, rendering information on a space from the image data to generate space information, recognizing the object in the space and mapping the object to the space information, recognizing the user in the space, and controlling the output unit on the basis of position information of the object and the user when a notification associated with the object occurs.

PRIORITY

This application is a National Phase Entry of International Application No. PCT/KR2016/013711, which was filed on Nov. 25, 2016, and claims priority to Korean Patent Application No. 10-2015-0172599, which was filed on Dec. 4, 2015, the contents of each of which are incorporated herein by reference.

TECHNICAL FIELD

Various embodiments of the present disclosure relate to a method for an electronic device to control audio output depending on space information and conditions of an object in a space in an electronic device, and a device thereof.

BACKGROUND ART

With the development of image processing technology, electronic devices can photograph a space using a camera and can identify an object existing in the space, thereby providing a user with notification information related to the object. In addition, with the development of audio processing technology, electronic devices can provide the user with the notification information related to the object using a sound.

DETAILED DESCRIPTION OF THE INVENTION

Technical Problem

As described above, in providing a user with information on an object identified by a camera using a sound, a conventional electronic device outputs the sound without considering the position of the user. Thus, the user pays attention to the electronic device having output the sound rather than the object that is an agent causing notification information, making it impossible to intuitively identify the position of the object. Further, after listening to all notification information transmitted via a sound, the user may know which object generated the notification information and thus waste unnecessary time figuring out which object caused the notification.

Various embodiments of the present disclosure provide a method and a device for processing an audio signal, in which an electronic device recognizes the positions of an object causing a notification and a user and adjusts a direction in which a sound physically proceeds and/or parameters for an audio signal to thereby provide an effect as if a sound is output from the object causing the notification.

Technical Solution

According to various embodiments of the present disclosure, an electronic device may include: a sensing unit configured to obtain image data and to sense an object and a user; an output unit configured to output a sound; and a processor configured to receive the image data from the sensing unit, to generate space information by rending information on a space from the image data, to recognize the object in the space, to map the object to the space information, and to control the output unit on the basis of location information on the object and the user if a notification associated with the object occurs.

Advantageous Effects

In various embodiments of the present embodiments, the output of a sound is controlled on the basis of location information of an object and location information of a user. Further, when a notification occurs, a sound is reflected on the object associated with the notification by changing an audio output direction and is transmitted to the user. Accordingly, the user may feel as if the sound is output from the object associated with the notification, that is, the object causing the notification. Thus, the user may be provided with intuitive User Experience (UX).

MODE FOR CARRYING OUT THE INVENTION

The expression “a first”, “a second”, “the first”, or “the second” used in various embodiments of the present disclosure may modify various components regardless of the order and/or the importance but does not limit the corresponding components. For example, a first user device and a second user device indicate different user devices although both of them are user devices. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element without departing from the scope of the present disclosure. [31] It should be understood that when an element (e.g., first element) is referred to as being (operatively or communicatively) “connected,” or “coupled,” to another element (e.g., second element), it may be directly connected or coupled directly to the other element or any other element (e.g., third element) may be interposer between them. In contrast, it may be understood that when an element (e.g., first element) is referred to as being “directly connected,” or “directly coupled” to another element (second element), there are no element (e.g., third element) interposed between them.

Hereinafter, an electronic device according to various embodiments will be described with reference to the accompanying drawings. In the present disclosure, the term “user” may indicate a person using an electronic device or a device (e.g., an artificial intelligence electronic device) using an electronic device.

FIG. 1illustrates an example of an electronic device according to various embodiments of the present disclosure.

Referring toFIG. 1, electronic devices may be divided into a standalone type11,12, or13and a docking station type14. The standalone-type electronic device11,12, or13may independently perform any function of an electronic device. The docking station-type electronic device14has two or more functionally separated electronic devices, which are combined into a single entity to perform any function of an electronic device. For example, the docking station-type electronic device14includes a main body14aand a driver14b, in which the main body14amay be mounted on a docking station (driver) and may be moved to a desired position.

Electronic devices may be classified into a stationary type11and a mobile type12,13, or14depending on mobility. The stationary-type electronic device11has no driver and thus cannot autonomously move. The mobile-type electronic device12,13, or14includes a driver and can autonomously move to a desired position. The mobile-type electronic device12,13, or14includes a wheel, a caterpillar, or a leg as a driver. Further, the mobile-type electronic device12,13, or14may include a drone.

FIG. 2illustrates an example of an electronic device according to various embodiments of the present disclosure.

An electronic device101may be configured in the form of a robot. The electronic device101may include a head portion190and a body portion193. The head portion190may be disposed above the body portion193. In one embodiment, the head portion190and the body portion193may be configured in shapes respectively corresponding to a human head and a human body. For example, the head portion190may include a front cover161corresponding to the shape of a human face. The electronic device101may include a display160disposed at the position corresponding to the front cover161. For example, the display160may be disposed inside the front cover161, in which case the front cover161may include a transparent material or semitransparent material. Alternatively, the front cover161may be an element capable of displaying a screen, in which case the front cover161and the display160may be configured as individual pieces of hardware. The front cover161may be at least one of various sensors for image sensing toward a direction in which interaction with a user is performed, at least one microphone for obtaining a sound, at least one speaker for audio output, an instrumental eye structure, and a display for outputting a screen, may display a direction using light or a temporary instrumental change, and may include at least one H/W or instrumental structure that faces the user when interaction with the user is performed.

The head portion190may further include a communication module170and a sensor171. The communication module170may receive a message from an external electronic device and may transmit a message to the external electronic device.

The sensor171may obtain information on an external environment. For example, the sensor171may include a camera, in which case the sensor171can photograph the external environment. The electronic device101may also identify the user on the basis of a photographing result. The sensor171may sense the proximity of the user to the electronic device101. The sensor171may sense the proximity of the user on the basis of proximity information or may sense the proximity of the user on the basis of a signal from another electronic device (e.g., a wearable device) used by the user. In addition, the sensor171may sense an action or position of the user.

A driver191may include at least one motor capable of moving the head portion190, for example, changing the direction of the head portion190. The driver191may be used to move or to instrumentally change another component. Further, the driver191may have a form enabling upward, downward, leftward, or rightward movements with respect to at least one axis, and the form of the driver191may be variously modified. A power supplier192may supply power used by the electronic device101.

A processor120may obtain a message wirelessly transmitted from another electronic device via the communication module170or may obtain a voice message via the sensor171. The processor120may include at least one message analysis module. The at least one message analysis module may extract or classify main content to be delivered to a recipient from the message generated by a sender.

A memory130is a repository capable of permanently or temporarily storing information related to providing a service to a user and may be disposed inside the electronic device101or may be disposed in a cloud or another server via a network. The memory130may store space information generated by the electronic device101or received from the outside. The memory130may store personal information for user authentication, property information about a method for providing service to the user, or information for identifying a relationship between various means capable of interacting with the electronic device101. Here, the information on the relationship may be changed by updating or learning information depending on the use of the electronic device101. The processor120is responsible for controlling the electronic device101and may functionally control the sensor171, an input/output interface150, the communication module170, and the memory130to provide services to the user. An information determination unit capable of determining information that the electronic device101can obtain may be included in at least a portion of the processor120or the memory130. The information determination unit may extract at least one or more pieces of data for a service from information obtained through the sensor171or the communication module170.

Although the electronic device101is configured in the form of a robot, which is merely an example, the electronic device101may be configured in any form without limitation.

In various embodiments of the present disclosure, the memory130may store instructions that enable the processor120, during an operation, to obtain an image and to output a message generated on the basis of an image analysis result, obtained by analyzing the obtained image, and additional information.

FIG. 3is a block diagram illustrating an electronic device and a network according to various embodiments of the present disclosure.

Referring toFIG. 3, the electronic device101may include a bus110, a processor120, a memory130, an input/output interface150, a display160, and a communication module170. In some exemplary embodiments, at least one of the components may be omitted, or other components may be further included in the electronic device101.

The bus110may include, for example, a circuit that connects the components110to170to each other and delivers communications (for example, control messages and/or data) between the components.

The processor120may include one or more of a Central Processing Unit (CPU), an Application Processor (AP), a Communication Processor (CP), a Graphic Processor (GP), a Multi-Chip Package (MCP), and an Image Processor (IP). The processor120may control, for example, at least one different component of the electronic device101, and/or may perform operations relating to communication or data processing.

The memory130may include a volatile and/or a nonvolatile memory. The memory130may store, for example, a command or data related to at least one different component of the electronic device101. According to one exemplary embodiment, the memory130may store software and/or a program140. The program140may include, for example, a kernel141, middleware143, an Application Programming Interface (API)145, and/or an application (or “app”)147. At least part of the kernel141, the middleware143, and the API145may be designated as an Operating System (OS).

The kernel141may control or manage system resources (for example, the bus110, the processor120, the memory130, or the like) used to perform an operation or function implemented, for example, in other programs (for example, the middleware143, the API145, or the application147). Further, the kernel141may provide an interface that allows the middleware143, the API145, or the application147to access individual components of the electronic device101to thereby control or manage system resources.

The middleware143may serve as a relay so that, for example, the API145or the application147communicates with the kernel141to exchange data.

Further, the middleware143may process one or more requests for operations received from the application147according to the priority thereof. For example, the middleware143may assign at least one application147a priority for using a system resource (for example, the bus110, the processor120, the memory130, or the like) of the electronic device101. For example, the middleware143may process the one or more requests for operations according to the priority assigned to the at least one application, thereby performing scheduling or load balancing for the one or more requests for operations.

The API145is, for example, an interface for the application147to control a function provided from the kernel141or the middleware143, and may include, for example, at least one interface or function (for example, a command) for file control, window control, image processing, or text control.

The input/output interface150may serve as an interface that delivers a command or data, which is input from, for example, a user or a different external device, to a different component(s) of the electronic device101. Further, the input/output interface150may output a command or data, which is received from a different component(s) of the electronic device101, to the user or to the different external device. The input/output interface150may include a touch input device, a voice input unit, various remote control devices, or the like. The input/output interface150may be at least one tool for providing a particular service to the user. For example, when information to be transmitted is a sound, the input/output interface150may be a speaker. When information to be transmitted is text or image content, the input/output interface150may be a display device. In addition, data that needs to be output in order to provide a service for the user who is far away from the electronic device101may be transmitted and output to at least one another electronic device through a communication module, in which the other electronic device may be a speaker or a different display device.

The display160may include, for example, a Liquid Crystal Display (LCD), a Light-Emitting Diode (LED) display, an Organic Light-Emitting Diode (OLED) display, a Micro-Electro-Mechanical Systems (MEMS) display, or an electronic paper display. The display160may display, for example, various types of content (for example, text, an image, a video, an icon, or a symbol) for the user. The display160may include a touch screen and may receive touch, gesture, proximity, or hovering input using, for example, an electronic pen or a body part of a user.

The communication module170may establish communication, for example, between the electronic device101and an external device (for example, a first external electronic device102, a second external electronic device104, or a server106). For example, the communication module170may be connected to a network162via wireless communication or wired communication to communicate with the external device (for example, the second external electronic device104or the server106). The communication module170is a tool capable of transmitting/receiving at least one data to/from another electronic device, which may communicate with another electronic device via at least one of (communication standards) Wireless Fidelity (Wi-Fi), ZigBee, Bluetooth, LTE, 3G, and IR protocols.

The wireless communication may use, for example, a cellular communication protocol, which may be, for example, at least one of Long-Term Evolution (LTE), LTE-Advanced (LTE-A), Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), Universal Mobile Telecommunications System (UMTS), Wireless Broadband (WiBro), and Global System for Mobile Communications (GSM). Further, the wireless communication may include, for example, short-range communication164. The short-range communication164may include, for example, at least one of Wi-Fi, Bluetooth, Near-Field Communication (NFC), and Global Navigation Satellite System (GNSS). The GNSS may include, for example, at least one of a Global Positioning System (GPS), a Global Navigation Satellite System (GLONASS), a BeiDou Navigation Satellite System, and Galileo, which is the European global satellite-based navigation system, depending on a use area or bandwidth. In the present document, “GPS” may be interchangeably used with “GNSS” hereinafter. The wire-based communication may include, for example, at least one of Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), Recommended Standard 232 (RS-232), and Plain Old Telephone Service (POTS). The network162may include a telecommunications network, which may be, for example, at least one of a computer network (for example, a Local Area Network (LAN) or Wide Area Network (WAN)), the Internet, and a telephone network.

The first and second external electronic devices102and104may each be a device of a type that is the same as, or different from, that of the electronic device101. According to one embodiment, the server106may include a group of one or more servers. According to various embodiments, all or some of the operations performed in the electronic device101may be performed in another electronic device or a plurality of electronic devices (for example, the electronic devices102and104or the server106). According to one embodiment, when the electronic device101needs to perform a function or service automatically or upon request, the electronic device101may request another electronic device (for example, the electronic device102or104, or the server106) to perform at least some functions related to the function or service, instead of, or in addition to, autonomously performing the function or service. The other electronic device (for example, the electronic device102or104, or the server106) may perform the requested functions or additional functions and may transmit the result thereof to the electronic device101. The electronic device101may provide the requested function or service using the same received result or by additionally processing the result. To this end, for example, cloud-computing, distributed-computing, or client-server-computing technologies may be used.

FIG. 4AandFIG. 4Bare block diagrams illustrating an electronic device according to various embodiments of the present disclosure.

The electronic device201may include, for example, all or part of the electronic device101illustrated inFIG. 3. The electronic device201may include one or more processors (for example, Application Processors (APs))210, a communication module220, a Subscriber Identification Module (SIM)224, a memory230, a sensor module240, an input device250, a display260, an interface270, an audio module280, a camera module291, a power management module295, a battery296, an indicator297, and a motor298.

The processors210may run, for example, an operating system or an application to control a plurality of hardware or software components that are connected to the processors210, and may perform various kinds of data processing and operations. The processors210may be configured, for example, as a System on Chip (SoC). According to one embodiment, the processors210may further include a Graphic Processing Unit (GPU) and/or an image signal processor. The processors210may include at least part (for example, a cellular module2) of the components illustrated inFIG. 4A. The processors210may load a command or data received from at least one of other components (for example, a nonvolatile memory) into a volatile memory to process the command or data, and may store various types of data in the nonvolatile memory.

The communication module220may have a configuration that is the same as, or similar to, that of the communication module170inFIG. 3. The communication module220may include, for example, a cellular module221, a Wi-Fi module223, a Bluetooth module225, a global navigation satellite system (GNSS) module227(for example, a GPS module, a GLONASS module, a BeiDou module, or a Galileo module), an NFC module228, and a Radio Frequency (RF) module229.

The cellular module221may provide, for example, a voice call, a video call, a text messaging service, or an Internet service through a communication network. According to one embodiment, the cellular module221may perform identification and authentication of the electronic device201in a communication network using a Subscriber Identity Module (SIM, for example, a SIM card)224. According to one embodiment, the cellular module221may perform at least some of the functions provided by the processors210. According to one embodiment, the cellular module221may include a Communication Processor (CP).

The Wi-Fi module223, the Bluetooth module225, the GNSS module227, and the NFC module228may each include, for example, a processor to process data transmitted and received via the respective modules. According to one embodiment, at least some (for example, two or more) of the cellular module221, the Wi-Fi module223, the Bluetooth module225, the GNSS module227, and the NFC module228may be included in one Integrated Chip (IC) or IC package.

The RF module229may transmit and receive, for example, a communication signal (for example, an RF signal). The RF module229may include, for example, a transceiver, a Power Amplifier (amp) Module (PAM), a frequency filter, a Low-Noise Amplifier (LNA), an antenna, or the like. According to another embodiment, at least one of the cellular module221, the Wi-Fi module223, the Bluetooth module225, the GNSS module227, and the NFC module228may transmit and receive an RF signal through a separate RF module.

The SIM224may include, for example, a card including a SIM and/or an embedded SIM, and may include unique identification information (for example, an Integrated Circuit Card Identifier (ICCID)) or subscriber information (for example, an International Mobile Subscriber Identity (IMSI)).

The memory230(for example, the memory130) may include, for example, an internal memory232or an external memory234. The internal memory232may include, for example, at least one of a volatile memory (for example, a Dynamic Random-Access Memory (DRAM), a Static RAM (SRAM), a Synchronous Dynamic RAM (SDRAM), or the like) and a nonvolatile memory (for example, a One-Time Programmable Read-Only Memory (OTPROM), a Programmable ROM (PROM), an Erasable And Programmable ROM (EPROM), an Electrically Erasable and a Programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory (for example, an NAND flash, an NOR flash, or the like), a hard drive, or a Solid-State Drive (SSD)).

The external memory234may further include a flash drive, for example, a Compact Flash (CF), a Secure Digital (SD), a micro Secure Digital (micro-SD), a mini Secure Digital (mini-SD), an extreme digital (xD), a Multi-Media Card (MMC), a memory stick, or the like. The external memory234may be functionally and/or physically connected to the electronic device201through various interfaces.

The sensor module240may measure, for example, physical quantities, or may detect the state of operation of the electronic device201and convert measured or detected information into an electrical signal. The sensor module240may include, for example, at least one of a gesture sensor240A, a gyro sensor240B, an atmospheric pressure sensor240C, a magnetic sensor240D, an acceleration sensor240E, a grip sensor240F, a proximity sensor240G, a color sensor240H (for example, a red, green, and blue (RGB) sensor), a biometric sensor240I, a temperature/humidity sensor240J, an illuminance sensor240K, and an ultraviolet (UV) sensor240M. Additionally or alternatively, the sensor module240may include, for example, an E-nose sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an iris sensor, and/or a fingerprint sensor. The sensor module240may further include a control circuit to control at least one or more sensors belonging thereto. In one embodiment, the electronic device201may further include a processor configured, as a part of the processors210or separately from the processors210, to control the sensor module240, thereby controlling the sensor module240while the processors210are in a sleep state.

The input device250may include, for example, a touch panel252, a (digital) pen sensor254, a key256, or an ultrasonic input device258. The touch panel252may be, for example, at least one of an electrostatic type, a pressure-sensitive type, an infrared type, and an ultrasonic type. Further, the touch panel252may further include a control circuit. The touch panel252may further include a tactile layer to provide a user with a tactile response.

The (digital) pen sensor254may, for example, be part of the touch panel or may include a separate recognition sheet. The key256may include, for example, a physical button, an optical key, or a keypad. The ultrasonic input device258may detect ultrasonic waves generated in an input tool through a microphone (for example, a microphone288) and may identify data corresponding to the detected ultrasonic waves.

The display260(for example, a display160) may include a panel262, a hologram device264, or a projector266. The panel262may include a configuration that is the same as, or similar to, that of the display160ofFIG. 1. The panel262may be configured, for example, to be flexible, transparent, or wearable. The panel262may be formed with the touch panel252in a single module. The hologram device264may display a three-dimensional image in the air using light interference. The projector266may project light onto a screen to display an image. The screen may be disposed, for example, inside or outside the electronic device201. According to one embodiment, the display260may further include a control circuit to control the panel262, the hologram device264, or the projector266.

The audio module280may bidirectionally convert, for example, a sound and an electrical signal. At least some components of the audio module280may be included, for example, in the input/output interface150illustrated inFIG. 3. The audio module280may process sound information input or output, for example, through a speaker282, a receiver284, earphones286, or the microphone288.

The camera module291is a device that takes, for example, a still image and a video. According to one embodiment, the camera module291may include one or more image sensors (for example, a front sensor or a rear sensor), a lens, an Image Signal Processor (ISP), or a flash (for example, an LED, a xenon lamp, or the like).

The power management module295may manage, for example, the power of the electronic device201. According to one embodiment, the power management module295may include a Power Management Integrated Circuit (PMIC), a charger Integrated Circuit (IC), or a battery or fuel gauge. The PMIC may have wire-based and/or wireless charging methods. The wireless charging methods may include, for example, a magnetic resonance method, a magnetic induction method, or an electromagnetic wave method, and may further include an additional circuit for wireless charging, such as a coil loop, a resonance circuit, or a rectifier. The battery gauge may measure, for example, the remaining battery charge, the charging voltage, the current, or temperature of the battery296. The battery296may include, for example, a rechargeable battery and/or a solar battery.

The indicator297may display the specific state of the electronic device201or a component thereof (for example, the processors210), which may be, for example, a booting state, a message state, or a charging state. The motor298may convert an electrical signal into mechanical vibrations, and may generate vibrations or a haptic effect. Although not shown, the electronic device201may include a processing device for supporting a mobile TV (for example, a GPU). The processing device for supporting the mobile TV may process media data, for example, in accordance with Digital Multimedia Broadcasting (DMB), Digital Video Broadcasting (DVB), or mediaFlo™ standards.

Referring toFIG. 4B, the processor210may be connected to an image recognition module241. In addition, the processor210may be connected to a behavior module244. The image recognition module241may include at least one of a two-dimensional (2D) camera242and a depth camera243. The image recognition module241may perform recognition on the basis of a photographing result and may transmit the recognition result to the processor210. The behavior module244includes at least one of: a facial expression motor245to display a facial expression of the electronic device101or to change the orientation of a face; a body pose motor245to change the pose of the body portion of the electronic device101, for example, the position of an arm, a leg, or a finger; and a moving motor247to move the electronic device101. The processor210may control at least one of the facial expression motor245, the body pose motor246, and the moving motor247to control the movement of the electronic device101configured in the form of a robot. The electronic device101may also include the components ofFIG. 4Bin addition to the components ofFIG. 4A.

Each of the above-described component elements of hardware according to the present disclosure may be configured with one or more components, and the names of the corresponding component elements may vary based on the type of electronic device. The electronic device according to various embodiments of the present disclosure may include at least one of the aforementioned elements. Some elements may be omitted or other additional elements may be further included in the electronic device. Also, some of the hardware components according to various embodiments may be combined into one entity, which may perform functions identical to those of the relevant components before the combination.

FIG. 5is a block diagram illustrating the software of an electronic device according to various embodiments of the present disclosure.

Referring toFIG. 5, the electronic device may include OS/system software501, middleware510, and an intelligent framework530.

The OS/system software501may distribute resources of the electronic device and may perform job scheduling and process processing. Also, the OS/system software may process data received from various hardware input units509. The hardware input units509may include at least one of a depth camera503, a 2D camera504, a sensor module505, a touch sensor506, and a microphone array507.

The middleware510may perform a function of the electronic device using data processed by the OS/system software501. According to one embodiment, the middleware510may include a gesture recognition manager511, a face detection/tracking/recognition manager512, a sensor information processing manager513, a dialog engine manager514, a voice synthesis manager515, a sound source tracking manager516, and a voice recognition manager517.

According to one embodiment, the face detection/tracking/recognition manager512may analyze an image captured by the 2D camera504to detect or track the position of the user's face, and may perform authentication through face recognition. The gesture recognition manager511may recognize a 3D gesture of the user by analyzing images captured by the 2D camera504and the depth camera503. The sound source tracking manager516may analyze a sound input through the microphone507and may track the input position of a sound source. The voice recognition manager517may analyze a voice input through the microphone507to recognize the inputted voice.

The intelligent framework530may include a multimodal fusion block531, a user pattern learning block532, and a behavior control block533. According to one embodiment, the multimodal fusion block531may collect and manage information processed in the middleware510. The user pattern learning block532may extract and learn significant information, such as the user's life pattern or preference, using the information in the multimodal fusion module531. The behavior control block533may provide information that the electronic device feeds back to the user as a motion of the electronic device, visual information, or audio information. That is, the behavior control block533may control a motor540of a driver to move the electronic device, may control a display such that a graphic object is displayed on the display550, or may control speakers561and562to output a sound.

A user model database521may store data learned by the electronic device via the intelligent framework530according to individual users. A behavior model database522may store data for behavior control of the electronic device. The user model database521and the behavior model database522may be stored in a memory of the electronic device, or may be stored in a cloud server via a network and may be shared with another electronic device502.

FIG. 6Aillustrates an interaction operation between an electronic device and an object according to various embodiments of the present disclosure.

Referring toFIG. 6A, the electronic device601may be disposed at the front of a space. Objects603and605may be disposed on the left of the electronic device601in the space, and objects609and611may be disposed on the right wall in the space. The objects603,605,609, and611may be things in the space. The objects603,605,609, and611may be things that are functionally connected to the electronic device601via wired or wireless communication or that are not functionally connected to the electronic device601. For example, the object603may be a telephone and may be functionally connected to the electronic device601. The object605may be a flowerpot and may not be functionally connected to the electronic device. The object609may be a clock and may be functionally connected to the electronic device. The object611may be a frame and may not be functionally connected to the electronic device.

The electronic device601may sense the space via a camera and may identify the position of a user and an object. The electronic device601may recognize the user and the object from an image captured through the camera, may extract location information on the user and the object, and may map the location information to space information. For example, referring to the captured image, the electronic device601may divide the structure of the space into front, rear, right, left, ceiling, or floor with respect to the electronic device601. For example, the electronic device may recognize that, on the basis of the electronic device, the object603is disposed in the left front portion off the space, the object605is disposed in the left rear portion of the space, the user607is located in the center of the space, the object609is disposed in the right rear portion of the space, and the object611is disposed in the right front portion of the space.

Alternatively, when the object is functionally connected to the electronic device601, the object may transmit information thereof to the electronic device. For example, the objects603and609may be functionally connected to the electronic device601and may transmit location information to the electronic device601.

The electronic device601may check the state of the object disposed in the space and may provide the user with a notification corresponding to the object. The notification may be provided as a sound through an output unit of the electronic device601. For example, when the electronic device601determines that the frame611is slanted, the electronic device601may provide the user with an audio notification reporting “The frame is leaning”. When the electronic device601determines that flowers in the flowerpot609are withering, the electronic device601may provide the user with an audio notification reporting “Please water the flowers”.

The electronic device601may control an audio output unit of the electronic device601on the basis of the location information on the user and the object. When providing the notification corresponding to the object, the electronic device601may control the audio output unit of the electronic device601to adjust an audio output direction, audio latency, a frequency band and/or volume per frequency band so that the user feels as if the sound is output from the object related to the notification. The electronic device601may identify the position of the user and the object in the space and may make the user feel as if the sound is output from the object on the basis of the location information on the user and the object triggering the notification. For example, when the frame611is leaning, the electronic device601may control the audio output unit to change an audio output direction610so that the user feels as if the sound is output from the position of the frame611. When a notification occurs due to the clock609, the electronic device601may control the audio output unit to change an audio output direction608so that the user feels as if the notification is output from the clock609. When a notification occurs due to the flowerpot605, the electronic device601may control the audio output unit to change an audio output direction606so that the user feels as if the notification is output from the flowerpot605.

Further, the electronic device601may control a driver of the electronic device601on the basis of the location information on the user and the object. When providing the notification corresponding to the object, the electronic device601may control the driver of the electronic device601to move the electronic device601to a particular position or to change the orientation of the electronic device601so that the user feels as if the sound is output from the object triggering the notification. The electronic device601may identify the position of the user and the object in the space and may control the driver of the electronic device601on the basis of the location information on the user and the object triggering the notification, thereby making the user feel as if the sound is output from the object. For example, when the frame611is leaning, the electronic device601may control the driver of the electronic device601to move the electronic device601or to change the orientation of the electronic device601, thus changing the audio output direction610, so that the user607feels as if the sound is output from the position of the frame611. When a notification occurs due to the clock609, the electronic device601may control the driver of the electronic device601to move the electronic device601or to change the orientation of the electronic device601, thus changing the audio output direction608, so that the user feels as if the notification is output from the clock. When a notification occurs due to the flowerpot605, the electronic device601may control the driver of the electronic device601to move the electronic device601or to change the orientation of the electronic device601, thus changing the audio output direction606, so that the user feels as if the notification is output from the flowerpot.

FIG. 6Billustrates an example in which an electronic device modifies an audio parameter according to various embodiments of the present disclosure.

FIG. 6Bshows the electronic device651, a user652, audio output devices653and654, and objects655,656, and657. The audio output devices653and654may be functionally connected to the electronic device651through wired or wireless communication and may include a driver. The orientation of the audio output devices603and2004may be changed by the drivers.

The electronic device651may obtain space information on where the electronic device651is located and may set a point where audio reflection may occur. The electronic device651may render the space information and may map the space information to the point where audio reflection may occur.

The objects655,656, and657may be things in the space. The objects655,656, and657may be things that are functionally connected to the electronic device651via wired or wireless communication or that are not functionally connected to the electronic device651. For example, the object655may be a flowerpot and may not be functionally connected to the electronic device651. The object656may be a clock and may be functionally connected to the electronic device651. The object657may be a frame and may not be functionally connected to the electronic device651.

The electronic device651may sense the space via a camera and may identify the position of the user652and the objects655,656, and657. The electronic device651may recognize the user652and the objects655,656, and657from an image captured through the camera, may extract location information on the user and the objects, and may map the location information to space information. For example, referring to the captured image, the electronic device651may divide the structure of the space into front, rear, right, left, ceiling, or floor with respect to the electronic device651. For example, the electronic device651may recognize that, on the basis of the electronic device651, the object655is disposed in the right middle portion of the space, the object656is disposed in the left front portion of the space, the object657is disposed in the left rear portion of the space, and the user652is located in the front rear portion of the space. When the objects655,656, and657are functionally connected to the electronic device651, the objects655,656, and657may transmit information thereof to the electronic device651. For example, the object656may be functionally connected to the electronic device651and may transmit location information to the electronic device651.

The electronic device651may identify the position of the objects655,656, and657and the position of the user652and may set an audio parameter so that the user652at the position feels as if a sound is output from the objects655,656, and657. As audio parameters, sound latency for left and right-channel sounds and/or volume per frequency band may be set.

For example, the electronic device651may identify the position of the flowerpot655in the space and the position of the user652, and may output a sound by setting, among the audio parameters, the sound latency for the left-channel sound to ‘0’, setting the sound latency for the right-channel sound to ‘3’, and setting the volume to ‘3’ so as to feel as if the sound is output from the flowerpot. Accordingly, the user may feel as if the sound is output from a left middle position respective to the user.

Alternatively, the electronic device651may generate an audio parameter (for example, the sound latency for the left-channel sound set to ‘0’, the sound latency for the right-channel sound set to ‘3’, and the volume set to ‘3’) and may transmit the audio parameter, along with audio data, to the audio output devices654and655so as to feel as if the sound is output from the flowerpot. The audio output devices653and654may set the sound latency for the left-channel sound to ‘0’, the sound latency for the right-channel sound set to ‘3’, and the volume set to ‘3’ on the basis of the audio parameter received from the electronic device651, thereby outputting a sound.

In another example, the electronic device651may identify the position of the clock656in the space and the position of the user652, and may output a sound by setting, among the audio parameters, the sound latency for the left-channel sound to ‘5’, setting the sound latency for the right-channel sound to ‘0’, and setting the volume to ‘2’ so as to feel as if the sound is output from the clock656. Accordingly, the user may feel as if the sound is output from a right middle position respective to the user.

Alternatively, the electronic device651may generate an audio parameter (for example, the sound latency for the left-channel sound set to ‘5’, the sound latency for the right-channel sound set to ‘0’, and the volume set to ‘2’) and may transmit the audio parameter, along with audio data, to the audio output devices654and655so as to feel as if the sound is output from the clock656. The audio output devices653and654may set the sound latency for the left-channel sound to ‘5’, the sound latency for the right-channel sound set to ‘0’, and the volume set to ‘2’ on the basis of the audio parameter received from the electronic device651, thereby outputting a sound.

In still another example, the electronic device651may identify the position of the frame657in the space and the position of the user652, and may output a sound by setting, among the audio parameters, the sound latency for the left-channel sound to ‘3’, setting the sound latency for the right-channel sound to ‘0’, and setting the volume to ‘4’ so as to feel as if the sound is output from the frame657. Accordingly, the user may feel as if the sound is output from a right middle position respective to the user.

Alternatively, the electronic device651may generate an audio parameter (for example, the sound latency for the left-channel sound set to ‘3’, the sound latency for the right-channel sound set to ‘0’, and the volume set to ‘4’) and may transmit the audio parameter, along with audio data, to the audio output devices654and655so as to feel as if the sound is output from the frame657. The audio output devices653and654may set the sound latency for the left-channel sound to ‘3’, the sound latency for the right-channel sound set to ‘0’, and the volume set to ‘4’ on the basis of the audio parameter received from the electronic device651, thereby outputting a sound.

FIG. 7is a block diagram schematically illustrating an electronic device according to various embodiments of the present disclosure.

Referring toFIG. 7, the electronic device may include a processor701, an input unit703, an output unit705, and a communication unit707.

The input unit703may include a camera, a sensor unit, and a microphone. The camera may photograph a space in which a user and an object are located and may output image data. The sensor unit may sense a motion of the user or the object and may output sensed data accordingly. Also, the sensor unit may measure the depth value of the user or object and may output sensed data accordingly. The microphone may receive a sound output from the electronic device and may output a signal accordingly. The electronic device may analyze a reflected sound input to the microphone and may generate a feedback signal on the sound output from the electronic device.

The processor701may include a room calibration module711, an object recognition module712, a listener recognition module713, and an output event generation unit714. The room calibration module711, the object recognition module712, the user recognition module713, and the output event generation module714may be stored as a program in a memory and may be loaded and executed as a command code in the processor701.

The room calibration module711may obtain information on a space in which the electronic device is disposed, using the image data and the sensor data received from the input unit. The electronic device may adjust a parameter of an audio output unit on the basis of the obtained information on the space.

The object recognition module712may recognize an object in the space where the electronic device is disposed. For example, when the space is a home, the object may be a TV, a sofa, a telephone, a flowerpot, furniture, a clock, or a PC, which may be present in the home. The object recognition module712may include a user recognition unit that recognizes at least one user.

The user recognition module713may identify the user using at least one biometric feature of the user. When there is no database having information on the identity of the user, the user recognition module713may classify the user by age, race, or gender using a physical feature of the user.

The communication unit707may receive data from the outside of the electronic device and may transmit the received data to the output event generation unit714. For example, when the electronic device receives a text message from the outside, the electronic device may transmit the text message to the output event generation unit714.

The output event generation module714may analyze information obtained via the input unit703, data received via the communication unit707, or information in the electronic device. When an output event is needed according to the analysis result, the output event generation module714may generate an output event appropriate for a situation and may transmit information necessary for event generation and information on a method for outputting generated information to the output unit705.

The output unit705may include a speaker driver to physically control a speaker and a speaker output unit to output the output event as a sound. The speaker driver or the speaker output unit may be formed with the electronic device in a single body or may be connected to the electronic device by a wired or wireless method. According to various embodiments, the output unit may not include a physical driver, and may output an audio notification by changing a parameter for audio output or by audio beamforming.

The output unit705may include a plurality of output units. The processor701may transmit an audio signal to be output via the plurality of output units to the output unit according to the generated event notification. An aimed direction may be determined depending on the positions of the plurality of output units, and thus the user may feel the directivity of an output sound.

FIG. 8is a flowchart illustrating a process in which an electronic device identifies a space and an object and provides a notification according to various embodiments of the present disclosure.

Referring toFIG. 8, the electronic device may obtain space information. For example, the electronic device may sense a space. The electronic device may photograph the space via a camera and may render captured image data into space information (801). Specifically, a room calibration module may render an image captured by the camera, thereby obtaining information on the space in which the electronic device is disposed. The electronic device may identify the structure of the space through the room calibration module. For example, the electronic device may render the space information, thereby recognizing that the space in which the electronic device is currently disposed includes front, rear, left and right sides, a ceiling, and a floor.

The electronic device may sense an object. The electronic device may recognize an object existing in the space after rendering the space information (803). For example, as a result of rendering the space information, when the space is recognized as a house, object recognition rate may be increased using spatial characteristics of the house. Specifically, the object recognition rate may be increased using information about an object that may exist in the house. When the object is recognized, the electronic device may map the position of the object to the space information obtained in801(805). The electronic device may determine the position of the object existing in the space by mapping the position of the object.

After recognizing the space and objects existing in the space, the electronic device may recognize a user existing in the space. When at least one user is recognized in the space, the electronic device may identify the user by referring to information that is already known. After recognizing the user, when the user is moving, the electronic device may track the position of the user (807). After the user is identified, the electronic device may map the position of the user to the space information. The electronic device may identify the user and may determine a direction in which the user looks. The electronic device may analyze the data captured by the camera to determine in which direction the user is currently looking, and may map the determined direction of the user to the space information. For example, the electronic device may determine whether the user looks at the electronic device, looks at the left wall or at the right wall, looks at the ceiling or the floor, or looks at the front wall, and may map the direction of the user to the space information. The electronic device may determine the optimal position where the user can listen to a sound, using direction information on the user mapped to the space information.

When mapping of the space information is completed, the electronic device may determine whether there is a notification received through a communication unit or a notification generated in the electronic device (809). When there is a notification, the electronic device may output the notification to the identified user via an audio output unit. When there is no notification, the electronic device may continue to perform the foregoing operations.

When there is a notification, the electronic device may generate a notification event. The electronic device may determine the notification event and may determine whether an object related to the notification event exists in a position where the user is currently located (811). Here, when an object related to the notification event is not currently found in the space, the electronic device may provide a general notification to the user (813). Here, the electronic device may transmit the notification via an output unit functionally connected to the electronic device. As a result of analyzing the notification event, when there is an object related to the notification in the space, the electronic device may find the position of the user and the position of the object from the rendered space information, and may determine an audio output direction and may adjust the position of a speaker on the basis of location information on the user and location information on the object (815). The electronic device may output the notification by adjusting the audio output direction of the audio output unit and the volume of a sound (817).

FIG. 9illustrates an example in which an electronic device renders a space according to various embodiments of the present disclosure.

FIG. 9shows the electronic device901disposed in a space. When the position of the electronic device901is changed, the electronic device901may receive space information on the space900of the position through an input unit. Specifically, the electronic device901may receive information about the space in which the electronic device901is disposed through a camera and a sensor device included in the electronic device901. Alternatively, the electronic device901may receive the space information via an external camera and sensor device functionally connected to the electronic device901.

The electronic device901may set a reflection plane in the space for sound positioning through reflection when recognizing the space, and may render the space on the basis of information on the reflection plane. For example, the electronic device901may divide the space into a front side907, a left side909, a right side903, a ceiling905, and a floor911on the basis of the current position of the electronic device901and may set each side as a reflection plane for reflecting a sound.

FIG. 10Aillustrates an embodiment in which an electronic device recognizes a user and monitors the position of the user. Referring toFIG. 10, the electronic device1001may recognize at least one user1003in a space1000. The electronic device1001may render space information and may then recognize the user1003in the space1000. When the user1003is recognized, the electronic device1001may identify the user1003by referring to a user model database. When the user1003is identified, the electronic device1001may map user information to the rendered space information.

When the user1003moves to a different position, the electronic device1001may identify the user1003and may recognize the new position of the user1003. The electronic device1001may determine the position of the user1003in the space and may recognize a direction in which the user1003looks. For example, the electronic device1001may render the space information, may identify the user1003, and may determine the position of the identified user1003to be a front left position. When rendering the space information, the electronic device1001may divide the space into a plurality of virtual areas, may determine a virtual area where the recognized user1003is located, and may determine the position of the user1003on the basis of the virtual area.

When the user1003is recognized, the electronic device1001may retrieve the face of the user1003and may determine in which direction the face of the user1003looks. When the direction of the face of the user1003is determined, the electronic device1001may map information on the direction in which the user1003looks to the space information.

FIG. 10Billustrates an example in which an electronic device recognizes and processes a plurality of users. Referring toFIG. 10B, an electronic device1001and a plurality of users1003,1005, and1007are in a space1000. The electronic device1001may render space information and may then recognize the plurality of users1003,1005, and1007in the space1000. When the plurality of users1003,1005, and1007is recognized, the electronic device1001may identify the plurality of users1003by referring to a user model database. When the plurality of users1003,1005, and1007is identified, the electronic device1001may map user information to the rendered space information.

The electronic device1001may determine the position of the plurality of users1003,1005, and1007in the space. When rendering the space information, the electronic device1001may divide the space into a plurality of virtual areas, may determine a virtual area where the recognized users are located, and may determine the position of the users on the basis of the virtual area. For example, the electronic device1001may render the space information, may identify the user1003, and may determine the position of the identified user1003to be a front left position. The electronic device1001may identify the user1005and may determine the position of the identified user1005to be a front left position. The electronic device1001may identify the user1007and may determine the position of the identified user1007to be a rear left position.

When a notification occurs with respect to the plurality of users1003,1005, and1007, the electronic device1001may provide a relevant notification to each user. For example, when a text message is received for the user1003, the electronic device1001may provide the user1003with an audio notification reporting that a text message is received. Here, the electronic device1001may set an audio parameter on the basis of location information on the user1003and may provide the user1003with the audio notification. The notification may be heard only by the user1003and may not be heard by the user1005and the user1007.

In another example, when an email is transmitted to the user1005, the electronic device1001may provide the user1005with an audio notification reporting that an email is received. The electronic device1001may set an audio parameter on the basis of location information on the user1005and may provide the user1005with the audio notification. The notification may be heard only by the user1005and may not be heard by the user1003and the user1007.

In still another example, when the user1007has an appointment, the electronic device1001may provide the user1007with a voice reporting that the user1007has an appointment. The electronic device1001may set an audio parameter on the basis of location information on the user1007and may provide the user1007with appointment information via a voice. The appointment information may be heard only by the user1007and may not be heard by the user1003and the user1005.

FIG. 11illustrates an example in which an electronic device monitors the state of an object according to various embodiments of the present disclosure.

Referring toFIG. 11, the electronic device1103and an object1105are disposed in a space1100.

The electronic device1103may recognize at least one object present in the space1100. When the object is recognized, the electronic device1103may map information on the recognized object to space information using any information that is accessible by the electronic device1103. The electronic device1103may determine the position of the recognized object in the space1103using the space information illustrated inFIG. 9.

The electronic device1103may separately set information about a situation where a notification is needed for a user according to the characteristics of the object. For example, when the electronic device1103recognizes the object1105in the space1100as a frame, the electronic device may recognize the basic characteristics of a frame. The basic characteristics of the frame may be set by the user or may be received from a database. The electronic device1103may recognize, on the basis of the basic characteristics of the frame, that a frame is generally disposed at a fixed position in a fixed direction. The electronic device1103may determine, as an abnormal situation, a situation where the orientation or position of a frame is changed even without any manipulation by the user and may set a notification determination criterion for a frame accordingly. That is, the electronic device1103may provide the user with a notification corresponding to the object1105on the basis of the basic characteristics of the object1105.

FIG. 12illustrates an example in which an output device functionally connected to an electronic device provides a notification according to various embodiments of the present disclosure. Referring toFIG. 12, the electronic device1203, an object1205, and output devices1207and1209are disposed in a space1200.

The object1205may be, for example, a frame. The output devices1207and1208may be functionally connected to the electronic device1203using wired or wireless communication. The output devices1207and1208may be, for example, a wireless speaker and may process audio signals of two channels.

The electronic device1203may recognize the object1205in the space and may provide a notification to a user according to the state of the object1205. The electronic device1203may generate the notification on the basis of the state of the object1205, and may provide the notification to the user via an audio output unit included in the electronic device1203or may transmit notification information to the output devices functionally connected to the electronic device1203. The output devices may output a sound by driving a speaker on the basis of the received notification information.

The electronic device1203may determine the position of the object in the space and may control the audio output unit on the basis of the position of the object. Specifically, the electronic device1203may control the audio output unit so that the user may feel as if the sound is generated from the object1205disposed in the space.

Alternatively, the electronic device1203may determine the position of the object in the space, may generate an audio parameter to control the audio output unit on the basis of the position of the object, and may transmit the audio parameter to the output devices. That is, the electronic device1203may transmit, to the output devices, the notification information generated on the basis of the state of the object and the audio parameter generated on the basis of the position of the object.

The output devices may control the speaker and an audio processor included in the output devices using the audio parameter received from the electronic device1203, thereby outputting a sound so that the user may feel as if the sound is generated from the object1205.

FIG. 13illustrates an example of an object sensing device functionally connected to an electronic device according to various embodiments of the present disclosure. The object sensing device may be included in the electronic device, or may be disposed outside the electronic device and may be functionally connected to the electronic device. The electronic device may receive information about an object from the functionally connected object sensing device.

Referring toFIG. 13, the electronic device1303and a user1305are positioned in a space1300. An object1307may be disposed outside the space1300. The object1307may be functionally connected to the electronic device1303via wireless communication. The object1307may be a CCTV having a wireless communication function and may be disposed in a porch outside the space1300.

The electronic device1303and the user1305may be inside the space1300. For example, the space1300may be a room in a house, and the electronic device1303and the user1305may be in the same room. The object1307functionally connectable to the electronic device1303may be positioned to a front wall relative to the position of the user1305in the space. For example, the object1307may be disposed beyond the front wall.

The electronic device1303may recognize an object in the space using various sensing tools, may determine the position of the object, and may map the position of the object to rendered space information. In various embodiments of the present disclosure, the electronic device1303may identify an object through a vision or audio sensing unit and may map the position of the object to the space information. Also, the electronic device1303may sense and recognize an object that is not in the same space as the electronic device1303through the object1307functionally connected to the electronic device1303. The object1307may be functionally connected to the electronic device1303using wired or wireless communication. The object1307may transmit location information thereof to the electronic device. For example, the object1307may determine the position thereof through an embedded GPS receiver or through an indoor positioning technique and may transmit the location information thereof to the electronic device1303. When an event occurs, the object1307may generate a notification, and may transmit notification information and the location information thereof to the electronic device1303.

The electronic device1303may generate audio information corresponding to the notification information on the basis of the notification information received from the object1307and may output the audio information through an audio output unit. Also, the electronic device1303may generate an audio parameter on the basis of the location information received from the object1307. The electronic device1303may control the audio output unit using the audio parameter and may change the direction of an output sound. As the direction of the audio output from the audio output unit is changed, the user may feel as if the sound is output in a direction in which the object1307is positioned.

For example, when the electronic device1303and the user1305are in the room and there is a visitor1309in the porch, the CCTV1307disposed in the porch may photograph the visitor and may generate notification information. The CCTV1307is functionally connected to the electronic device1303via a Wi-Fi network and may transmit the notification information and location information thereof to the electronic device1303in the room.

The electronic device1303may map the location information on the CCTV received from the object1307to the space information. The electronic device1303may identify the position of the user1305and a direction in which the user1305looks through a camera and may generate an audio parameter on the basis of the identified position of the user1305and the location information on the CCTV received from the object1307. The electronic device1303may generate a notification message corresponding to the notification information received from the object1307and may control the audio output unit using the audio parameter to output a sound.

For example, the electronic device1303may control the audio output unit so that the user1305may feel as if the sound reporting that there is the visitor1309is output from the front wall1310beyond which the porch is positioned.

FIG. 14illustrates an example of an object that may exist in a house.

FIG. 14shows various objects existing in a house. The objects may be divided into a communication object that has a communication function and is functionally connectable to an electronic device and a general object having no communication function. The communication object may be an Internet of Things (IoT) device. For example, the communication object may be a burglar alarm device security system1401, a window control1402, a Set-Top Box (STB)1403, a remote controller1404, a door control1405, a motion sensor1406, an environmental control1407, a Heating, Ventilation and Air Conditioning (HVAC) control1408, or a light control1409. The general object having no communication function may be, for example, frames1410and1411.

The communication object may transmit location information thereof to the electronic device. The communication object may include a GPS reception device, may measure the position thereof using GPS information or an indoor positioning technique, and may transmit measured location information to the electronic device. Alternatively, the location information on the communication object may be set by a user.

Location information on the general object may be set by the user or may be set by the electronic device with reference to space information rendering and the location information on the communication object.

When the location information on the communication object and the location information on the general object are determined, the electronic device may map the positions of the communication object and the general object to rendered space information.

FIG. 15is a flowchart illustrating a process in which an electronic device recognizes an object and generates a notification when an event occurs according to various embodiments of the present disclosure.

Referring toFIG. 15, the electronic device may be functionally connected to at least one object through a network. For example, the electronic device1501may be functionally connected to a first object1502and a second object1503via wired or wireless communication. While communicating with the electronic device1501, the first object1502and the second object1503may transmit location information (e.g., GPS information) on the first object1502and the second object1503to the electronic device1501in a session negotiation process (1504). The electronic device1501may set and store the relative coordinates of the first object1502and the second object1503on the basis of the location information on the first object1502and the second object1503and current location information on the electronic device1501.

When an event occurs to the first object1502(1505), the electronic device1501may determine the position of a user (1506), may generate an audio parameter on the basis of location information on the user, and may control an audio output unit (1507). The electronic device1501may change an audio output direction using the audio parameter and may provide notification information to the user.

When an event occurs to the second object1503(1508), the electronic device1501may determine the position of a user (1509), may generate an audio parameter on the basis of location information on the user, and may control the audio output unit (1510). The electronic device1501may change an audio output direction using the audio parameter and may provide notification information to the user.

FIGS. 16 and 17illustrate examples in which an electronic device recognizes the state of an object and provides a notification to a user according to various embodiments of the present disclosure.

FIG. 16shows an electronic device1602and an object1603in a space1601. The object may be disposed on a front wall relative to the electronic device. Front, rear, top, bottom, left, and right sides of the space may be reflection planes that reflect a sound.

The electronic device1602may analyze an image captured by a camera to recognize the object1603and may determine whether the object is in a normal state. When it is determined that the type of the recognized object1603is a frame, the electronic device1602may distinguish the normal state of a frame from an abnormal state of a frame by referring to a database. Alternatively, when it is determined that the recognized object1603is a frame, if the position of the frame is not changed for a predetermined time, the electronic device1602may determine that the frame1603is in the normal state. When the position of the frame is changed, the electronic device1602may determine that the frame is in the abnormal state.

FIG. 17shows an electronic device1702and an object1703in a space1701. The object1703may be disposed on a front wall relative to the electronic device1702and may be disposed in an abnormal state. For example, the object1703may be a frame, which may be disposed on the front wall in a slanted state.

Front, rear, top, bottom, left, and right sides of the space1701may be reflection planes that reflect sound. The electronic device1702may analyze an image captured by a camera to recognize the object1703and may determine whether the object1703is in the abnormal state. When it is determined that the recognized object1703is a frame, the electronic device1702may distinguish a normal state of a frame from an abnormal state of a frame by referring to a database. When it is determined that the object1703is in the abnormal state, the electronic device1702may generate a notification event corresponding to the object1703and generate notification information. Further, the electronic device1702may generate an audio parameter on the basis of location information on a user1704and location information on the object1703and may control an audio output unit using the audio parameter. The electronic device1702may control the audio output unit to change an audio output direction, making the user1704feel as if a sound is output from the position of the object1703.

That is, the electronic device1702may generate a notification on the basis of state information on the object1703and may control the audio output unit on the basis of the location information on the object1703and the location information on the user1704.

FIGS. 18 and 19illustrate examples in which an electronic device controls an audio output unit on the basis of the positions of an object and a user according to various embodiments of the present disclosure.

FIG. 18shows an electronic device1801, an object1803, a user1802, and audio output devices1804and1805. The audio output devices1804and1805may be functionally connected to the electronic device1801via wired/wireless communication.

The electronic device1801may obtain space information on where the electronic device1801is located and may set a point at which sound reflection may occur. The electronic device1801may render the space information and may map the point where sound reflection may occur to the space information.

The electronic device1801may recognize the object1803in the space, may determine the type of the recognized object1803, and may determine whether the recognized object1803is currently in a condition where a notification is needed. The electronic device1801may check the state of the recognized object1803, and may identify the position of the user1802when it is determined that the recognized object1803is in a condition where a notification is needed. When the position of the user1802is identified, the electronic device1801may determine an audio output direction to generate an audio parameter on the basis of user location information on the user1802and location information on the object1803, and may control an audio output unit or may control a driver of the electronic device1801. Alternatively, the electronic device1801may transmit the generated audio parameter to the audio output devices1804and1805.

The electronic device1801may generate notification information corresponding to the object1803and may output the generated notification information through the audio output unit or the audio output devices1804and1805. Here, an output sound may be reflected on reflection walls1806and1807of the space and may be transmitted to the user1802, and the user1802may feel as if the sound is output in the direction of the object1803about which a notification is needed.

FIG. 19shows an electronic device1901, an object1903, a user1902, and audio output devices1904and1905. The audio output devices1904and1905may be functionally connected to the electronic device1901through wired/wireless communication and include a driver. The direction of the audio output devices1904and1905may be changed by the driver.

The electronic device1901may obtain space information on where the electronic device1901is located and may set a point at which sound reflection may occur. The electronic device1901may render the space information and may map the point where sound reflection may occur to the space information.

The electronic device1901may recognize the object1903in the space, may determine the type of the recognized object1903, and may determine whether the recognized object1903is currently in a condition where a notification is needed. The electronic device1901may check the state of the recognized object1903, and may identify the position of the user1902when it is determined that the recognized object1903is in a condition where a notification is needed. When the position of the user1902is identified, the electronic device1901may determine an audio output direction to generate an audio parameter on the basis of user location information on the user1902and location information on the object1903, and may control an audio output unit or may control a driver of the electronic device1901. The electronic device1901may set a reflection point1908on reflection walls1906and1907in the space on the basis of the user location information and the location information on the object1903. Alternatively, the electronic device1901may transmit the generated audio parameter to the audio output devices1904and1905. The audio output devices1904and1905may include a driver and may control the driver using the audio parameter transmitted from the electronic device1901. Further, the audio output devices1904and1905may include an audio processor, may process a sound in the audio processor using the audio parameter transmitted from the electronic device1901, and may drive the speaker to output a sound.

The electronic device1901may generate notification information corresponding to the object and may output the generated notification information through the audio output unit or the audio output devices. Here, an output sound may be reflected at the reflection point in the space and may be transmitted to the user1902, and the user1902may feel as if the sound is output in the direction of the object1903about which a notification is needed.

FIG. 20illustrates an example in which an electronic device provides an audio notification with the characteristics of an object reflected according to various embodiments of the present disclosure.

FIG. 20shows an electronic device2001, a user2002, audio output devices2003and2004, and objects2005,2006, and2007. The audio output devices2003and2004may be functionally connected to the electronic device2001through wired/wireless communication and may include a driver. The direction of the audio output devices2003and2004may be changed by the driver.

The electronic device2001may obtain space information on where the electronic device2001is located and may set a point at which sound reflection may occur. The electronic device2001may render the space information and may map the point where sound reflection may occur to the space information.

The objects2005,2006, and2007may be things in the space. The objects2005,2006, and2007may be things that are functionally connected to the electronic device2001via wired or wireless communication or that are not functionally connected to the electronic device2001. For example, the object2005may be a flowerpot and may not be functionally connected to the electronic device2001. The object2006may be a clock and may be functionally connected to the electronic device2001. The object2007may be a frame and may not be functionally connected to the electronic device2001.

The electronic device2001may sense the space via a camera and may identify the position of the user2002and the objects2005,2006, and2007. The electronic device2001may recognize the user2002and the objects2005,2006, and2007from an image captured through the camera, may extract location information on the user and the objects, and may map the location information to space information. For example, referring to the captured image, the electronic device2001may divide the structure of the space into front, rear, right, left, ceiling, or floor with respect to the electronic device2001. For example, the electronic device2001may recognize that, on the basis of the electronic device2001, the object2005is disposed in the right middle portion of the space, the object2006is disposed in the left front portion of the space, the object2007is disposed in the left rear portion of the space, and the user2002is located in the front rear portion of the space. When the objects2005,2006, and2007are functionally connected to the electronic device2001, the objects2005,2006, and2007may transmit information thereof to the electronic device2001. For example, the object2006may be functionally connected to the electronic device2001and may transmit location information to the electronic device2001.

The electronic device2001may identify the state and characteristics of the objects disposed in the space and may provide the user2002with a notification corresponding to the objects. The notification may be provided as a sound through an output unit of the electronic device2001. The electronic device2001may provide the notification by modifying the notification into a voice appropriate for the characteristics of the objects in order to maximize the effect of providing the notification.

For example, when the electronic device2001photographs the flowerpot2005in the space to obtain image data, analyzes the obtained data, and determines that flowers in the flowerpot2005wilted, the electronic device2001may provide the user2002with an audio notification reporting “Please water me.” The electronic device2001may analyze the characteristics of the photographed flowerpot2005, may classify a characteristic as female, and may provide the notification with a female voice in order to maximize the effect of the notification. The electronic device2001may refer to an external database or user setup data in order to identify the characteristics of the objects. For example, the external database has information about what kind of voice is to be provided for the characteristics of various objects, and the electronic device2001may refer to the external database in providing a notification and may provide a notification with a corresponding voice. Alternatively, when there is data preset by a user, the electronic device2001may refer to user setup data and may provide a notification with a corresponding voice. Alternatively, when image date obtained from a photographed object includes a person, the electronic device2001may recognize the person and may change and provide a notification corresponding to a voice appropriate for the gender or age of the identified person. For example, when an identified person is a woman in her 20s, the electronic device2001may provide a notification with a voice of a woman in her 20s. Further, when a characteristic of a photographed object is identified as a man in his 30s, the electronic device2001may provide a notification with a voice of a man in his 30s.

Further, the electronic device2001may identify the position of the flowerpot2005in the space and the position of the user2002and may adjust an audio output direction, the latency and frequency band of a sound, or the volume per frequency by controlling the audio output unit of the electronic device2001in order to feel as if a sound is output from the flowerpot. Alternatively, the electronic device2001may control a driver of the electronic device2001and may change the orientation of the electronic device2001in order to feel as if a sound is output from the flowerpot. Alternatively, the electronic device2001may generate an audio parameter for adjusting an audio output direction or the latency and the frequency band of a sound and may transmit the generated audio parameter to the audio output devices2004and2005in order to feel as if a sound is output from the flowerpot. The audio output devices2004and2005may include a driver and may control the driver using the audio parameter transmitted from the electronic device2001. Further, the audio output devices2004and2005may include an audio processor, may process a sound in the audio processor using the audio parameter transmitted from the electronic device2001, and may drive a speaker to output the sound.

In another example, when the electronic device2001photographs the clock2006in the space to obtain image data, analyzes the obtained data, and determines that the clock2006displays the incorrect time, the electronic device2001may provide the user with an audio notification reporting “Please correct the time.”

The electronic device2001may analyze the characteristics of the photographed clock2005, may classify a characteristic as male, and may provide the notification with a male voice in order to maximize the effect of the notification. The electronic device2001may refer to an external database or user setup data in order to identify the characteristics of the objects. For example, the external database has information about what kind of voice is to be provided for the characteristics of various objects, and the electronic device2001may refer to the external database in providing a notification and may provide a notification with a corresponding voice. Alternatively, when there is data preset by the user, the electronic device2001may refer to user setup data and may provide a notification with a corresponding voice. Alternatively, when image date obtained from a photographed object includes a person, the electronic device2001may recognize the person and may change and provide a notification corresponding to a voice appropriate for the gender or age of the identified person. For example, when an identified person is a woman in her 20s, the electronic device2001may provide a notification with a voice of a woman in her 20s. Further, when a characteristic of a photographed object is identified as a man in his 30s, the electronic device2001may provide a notification with a voice of a man in his 30s.

Further, the electronic device2001may identify the position of the clock2006in the space and the position of the user and may adjust an audio output direction, the latency and frequency band of a sound, or the volume per frequency by controlling the audio output unit of the electronic device2001in order to feel as if a sound is output from the clock. Alternatively, the electronic device2001may control the driver of the electronic device2001and may change the orientation of the electronic device2001in order to feel as if a sound is output from the clock. Alternatively, the electronic device2001may generate an audio parameter for adjusting an audio output direction or the latency and the frequency band of a sound and may transmit the generated audio parameter to the audio output devices2004and2005in order to feel as if a sound is output from the clock. The audio output devices2004and2005may include a driver and may control the driver using the audio parameter transmitted from the electronic device2001. Further, the audio output devices2004and2005may include an audio processor, may process a sound in the audio processor using the audio parameter transmitted from the electronic device2001, and may drive a speaker to output the sound.

In still another example, when the electronic device2001photographs the frame2007in the space to obtain image data, analyzes the obtained data, and determines that the frame2005is slanted, the electronic device2001may provide the user with an audio notification reporting “Please straighten me.”

The electronic device2001may analyze the characteristics of the photographed frame2007, may identify that a male face is shown in the frame, and may provide the notification with a male voice in order to maximize the effect of the notification. The electronic device2001may refer to an external database or user setup data in order to identify the characteristics of the objects. For example, the external database has information about what kind of voice is to be provided for the characteristics of various objects, and the electronic device2001may refer to the external database in providing a notification and may provide a notification with a corresponding voice. Alternatively, when there is data preset by the user2002, the electronic device2001may refer to user setup data and may provide a notification with a corresponding voice. Alternatively, when image date obtained from a photographed object includes a person, the electronic device2001may recognize the person and may change and provide a notification corresponding to a voice appropriate for the gender or age of the identified person. For example, when an identified person is a woman in her 20s, the electronic device2001may provide a notification with a voice of a woman in her 20s. Further, when a characteristic of a photographed object is identified as a man in his 30s, the electronic device2001may provide a notification with a voice of a man in his 30s.

Further, the electronic device2001may identify the position of the frame2007in the space and the position of the user2002and may adjust an audio output direction, the latency and frequency band of a sound, or the volume per frequency by controlling the audio output unit of the electronic device2001in order to feel as if a sound is output from the frame. Alternatively, the electronic device2001may control the driver of the electronic device2001and may change the orientation of the electronic device2001in order to feel as if a sound is output from the clock. Alternatively, the electronic device2001may generate an audio parameter for adjusting an audio output direction or the latency and the frequency band of a sound and may transmit the generated audio parameter to the audio output devices2004and2005in order to feel as if a sound is output from the clock. The audio output devices2004and2005may include a driver and may control the driver using the audio parameter transmitted from the electronic device2001. Further, the audio output devices2004and2005may include an audio processor, may process a sound in the audio processor using the audio parameter transmitted from the electronic device2001, and may drive a speaker to output the sound.

As described above, the electronic device2001may analyze the characteristics of a photographed object and may provide a notification by converting a voice on the basis of the characteristics of the photographed object. Here, the characteristics may be, for example, determining the gender and age of the object, and the electronic device2001may provide the notification by converting a voice corresponding to the determined gender and age of the object.

Each of the components of the electronic device according to the present disclosure may be implemented by one or more components and the name of the corresponding component may vary depending on a type of the electronic device. In various embodiments, the inspection apparatus may include at least one of the above-described elements. Some of the above-described elements may be omitted from the electronic device, or the inspection apparatus may further include additional elements. Further, some of the components of the electronic device according to the various embodiments of the present disclosure may be combined to form a single entity, and thus, may equivalently execute functions of the corresponding elements prior to the combination.

The term “module” as used herein may, for example, mean a unit including one of hardware, software, and firmware or a combination of two or more of them. The “module” may be interchangeably used with, for example, the term “unit”, “logic”, “logical block”, “component”, or “circuit”. The “module” may be the smallest unit of an integrated component or a part thereof. The “module” may be the smallest unit that performs one or more functions or a part thereof. The “module” may be mechanically or electronically implemented. For example, the “module” according to the present disclosure may include at least one of an Application-Specific Integrated Circuit (ASIC) chip, a Field-Programmable Gate Arrays (FPGA), and a programmable-logic device for performing operations which has been known or are to be developed hereinafter.

According to various embodiments, at least some of the devices (for example, modules or functions thereof) or the method (for example, operations) according to the present disclosure may be implemented by a command stored in a computer-readable storage medium in a programming module form. When the command is executed by one or more processors (for example, the processor120), the one or more processors may execute a function corresponding to the command. The computer-readable storage medium may, for example, be the memory130.

The programming module according to the present disclosure may include one or more of the aforementioned components or may further include other additional components, or some of the aforementioned components may be omitted. Operations executed by a module, a programming module, or other component elements according to various embodiments of the present disclosure may be executed sequentially, in parallel, repeatedly, or in a heuristic manner. Furthermore, some operations may be executed in a different order or may be omitted, or other operations may be added.

According to various embodiments of the present disclosure, a storage medium stores commands, wherein the commands are set for at least one processor to perform at least one operation when executed by the at least one processor, and the at least one operation may include: obtaining an image; and outputting a message generated on the basis of an image analysis result of analyzing the obtained image and additional information.

Various embodiments disclosed herein are provided merely to easily describe technical details of the present disclosure and to help the understanding of the present disclosure, and are not intended to limit the scope of the present disclosure. Therefore, it should be construed that all modifications and changes or modified and changed forms based on the technical idea of the present disclosure fall within the scope of the present disclosure.