Method for producing haptic signal and electronic device supporting the same

An electronic device includes a memory, a communication interface capable of performing communication, and a processor electrically connected to the memory and the communication interface. The processor is configured to extract a first audio signal to be output through a speaker, based on audio data received from an external electronic device and/or stored in the memory, to generate a second audio signal obtained by removing the first audio signal from the audio data, and to generate a haptic signal for outputting a haptic effect, based on the second audio signal.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit under 35 U.S.C. § 119(a) of Korean patent application assigned Serial number 10-2016-0162161, filed on Nov. 30, 2016 in the Korean Intellectual Property Office, the entire disclosure of which is hereby incorporated herein by reference.

FIELD

The present disclosure relates to a method of generating a haptic signal(s), which a user is able to sense, and/or an electronic device for generating a haptic signal(s).

BACKGROUND

Thanks to the increase in the resolution and the operating speed of an electronic device and the improvement of the performance of a graphic processing device, a device outputting virtual reality or a 3-dimensional stereoscopic image, which only a large-size device drives, has been miniaturized and lightened through a smart glass, a head mount device (hereinafter is referred to as a “HMD”), or the like.

For example, an HMD may be mounted on a user's head and may provide the effect of displaying a large screen in front of the user's eyes (e.g., the user's gaze). The HMD may be used to output a stereoscopic image. An augmented reality (AR) service or a virtual reality (VR) service using the HMD or a smartphone has been developed.

In addition, an electronic device may provide the user with haptic effects (or tangible effects) (e.g., vibration, motion, and the like) as well as visual effects and auditory effects, thereby improving the sense of presence.

SUMMARY

A conventional haptic device (or a device of tangible style) (e.g., HMD) may output a haptic effect (e.g., vibration generated depending on a specified timing signal) of a simple type when outputting a stereoscopic image. In this case, since the user feels a simple vibration, the user does not feel the sense of presence by various senses.

In accordance with an example aspect of the present disclosure, an electronic device includes a memory, a communication interface (including communication interface circuitry) capable of performing communication, and a processor electrically connected to the memory and the communication interface. The processor is configured: to extract a first audio signal to be output through a speaker, based at least on audio data received from an external electronic device and/or stored in the memory, to generate a second audio signal obtained by removing the first audio signal from the audio data, and to generate a haptic signal for outputting a haptic effect, based at least on the second audio signal.

According to various example embodiments of the present disclosure, a haptic signal producing method and/or an electronic device supporting the same may produce a haptic signal(s) based at least on a signal (e.g., a wind noise) that has been removed as a noise signal.

According to various example embodiments of the present disclosure, a haptic signal producing method and an electronic device supporting the same may produce a haptic signal by reflecting various features such as the intensity, the frequency, or the like of a wind noise, and then may provide the user with a haptic effect of various forms. Accordingly, the user feels the sense of presence at a point in time when the user captures an image by using a camera or a camcorder, when the user admires the image.

According to various embodiments of the present disclosure, a haptic signal producing method and an electronic device supporting the same may produce and transmit a haptic signal, which allows a user to feel the sense of presence, by using a circuit executing an existing function to reduce a wind noise.

DETAILED DESCRIPTION

Hereinafter, various example embodiments of the present disclosure will be described with reference to the accompanying drawings. Accordingly, those of ordinary skill in the art will recognize that modification, equivalent, and/or alternative on the various embodiments described herein can be variously made without departing from the scope and spirit of the present disclosure. With regard to description of drawings, similar components may be marked by similar reference numerals.

In the disclosure disclosed herein, the expressions “have”, “may have”, “include” and “comprise”, or “may include” and “may comprise” used herein indicate existence of corresponding features (for example, elements such as numeric values, functions, operations, or components) but do not exclude presence of additional features.

The terms, such as “first”, “second”, and the like used herein may refer to various elements of various embodiments of the present disclosure, but do not limit the elements. For example, such terms are used only to distinguish an element from another element and do not limit the order and/or priority of the elements. For example, a first user device and a second user device may represent different user devices irrespective of sequence or importance. For example, without departing the scope of the present disclosure, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

It will be understood that when an element (for example, a first element) is referred to as being “(operatively or communicatively) coupled with/to” or “connected to” another element (for example, a second element), it can be directly coupled with/to or connected to the other element or an intervening element (for example, a third element) may be present. In contrast, when an element (for example, a first element) is referred to as being “directly coupled with/to” or “directly connected to” another element (for example, a second element), it should be understood that there are no intervening element (for example, a third element).

An electronic device according to various embodiments of the present disclosure may include at least one of smartphones, tablet personal computers (PCs), mobile phones, video telephones, electronic book readers, desktop PCs, laptop PCs, netbook computers, workstations, servers, personal digital assistants (PDAs), portable multimedia players (PMPs), MP3 players, mobile medical devices, cameras, and wearable devices. According to various embodiments of the present disclosure, the wearable devices may include accessories (for example, watches, rings, bracelets, ankle bracelets, glasses, contact lenses, or head-mounted devices (HMDs)), cloth-integrated types (for example, electronic clothes), body-attached types (for example, skin pads or tattoos), or implantable types (for example, implantable circuits).

In some embodiments of the present disclosure, the electronic device may be one of home appliances. The home appliances may include, for example, at least one of a digital video disk (DVD) player, an audio, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washing machine, an air cleaner, a set-top box, a home automation control panel, a security control panel, a TV box (for example, Samsung HomeSync™, Apple TV™, or Google TV™), a game console (for example, Xbox™ or PlayStation™), an electronic dictionary, an electronic key, a camcorder, or an electronic panel.

In another embodiment of the present disclosure, the electronic device may include at least one of various medical devices (for example, various portable medical measurement devices such as a blood glucose meter, a heart rate measuring device, a blood pressure measuring device, and/or a body temperature measuring device), a magnetic resonance angiography (MRA), a magnetic resonance imaging (MRI) device, a computed tomography (CT) device, a photographing device, and an ultrasonic device), a navigation system, a global navigation satellite system (GNSS), an event data recorder (EDR), a flight data recorder (FDR), a vehicular infotainment device, electronic devices for vessels (for example, a navigation device for vessels and a gyro compass), avionics, a security device, a vehicular head unit, an industrial or home robot, an automatic teller's machine (ATM) of a financial company, a point of sales (POS) of a store, or an internet of things (for example, a bulb, various sensors, an electricity or gas meter, a spring cooler device, a fire alarm device, a thermostat, an electric pole, a toaster, a sporting apparatus, a hot water tank, a heater, and a boiler).

According to some embodiments of the present disclosure, the electronic device may include at least one of a furniture or a part of a building/structure, an electronic board, an electronic signature receiving device, a projector, or various measurement devices (for example, a water service, electricity, gas, or electric wave measuring device). In various embodiments of the present disclosure, the electronic device may be one or a combination of the aforementioned devices. The electronic device according to some embodiments of the present disclosure may be a flexible electronic device. Further, the electronic device according to an embodiment of the present disclosure is not limited to the aforementioned devices, but may include new electronic devices produced due to the development of technologies.

Hereinafter, electronic devices according to example embodiments of the present disclosure will be described with reference to the accompanying drawings. The term “user” used herein may refer to a person who uses an electronic device or may refer to a device (for example, an artificial electronic device) that uses an electronic device.

FIG. 1illustrates an image output system, according to various example embodiments.

Referring toFIG. 1, an image output system100may include a camera device110, a first electronic device120, a server130, a database131, and a haptic device150, which may be connected through a network160.

The camera device110may include a lens, an image sensor, and the like and may capture an image of the surrounding area. The camera device110may collect video data and audio data at the same time. In various example embodiments, the camera device110may include a sensor module (e.g., a GPS, a gyroscope, an accelerometer, a geomagnetic sensor, or the like). The camera device110may collect motion data (e.g., movement, slope, direction, or the like) while the video data or the audio data is collected.

The camera device110may store the collected video data, the collected audio data, and/or the collected motion data in internal memory, and/or may stream the collected video data, the collected audio data, or the collected motion data to an external device.

According to various example embodiments, the camera device110may include a plurality of image sensors (e.g., a first image sensor and a second image sensor) and a plurality of lenses through which light is concentrated on an image sensor. For example, the first image sensor and a first lens may capture a first image having angle of view of 180 degrees or more in a first direction, and the second image sensor and a second lens may capture a second image having angle of view of 180 degrees or more in a second direction opposite to the first direction. Accordingly, the camera device110may obtain an image of 360 degrees.

According to various example embodiments, the camera device110may include one or more microphones (or a microphone array). The camera device110may collect an ambient sound through a microphone to generate audio data. The audio data may include various sounds such as a sound output together with an image, an ambient sound, a noise sound, and the like. In the case where the camera device110includes the plurality of microphones, the plurality of microphones may be disposed to be spaced apart by a specified distance. The audio data collected through a plurality of microphones may be used to analyze the feature of a sound.

According to various example embodiments, the camera device110may include a processor (e.g., AP or MCU) for an operation to collect video data, audio data, and/or motion data, and may include a communication interface (including communication interface circuitry) that transmits the collected data to an external device.

The first electronic device120may generate a stereoscopic image (e.g., an image of 360 degrees) based on information (video data, audio data, motion data, or the like) collected through the camera device110. For example, the first electronic device120may receive a plurality of images corresponding to other gazes to perform an operation such as stitching for generating an equi-rectangular projection (ERP) image, texture mapping, or the like.

According to various example embodiments, the camera device110may be mounted in the first electronic device120or may be a camera module included in the first electronic device120.

According to various example embodiments, the first electronic device120may generate haptic information based on the collected audio data. The first electronic device120may produce a haptic signal(s) to be output by the haptic device150, based on the generated haptic information.

According to various example embodiments, the first electronic device120may include a communication interface, including communication interface circuitry, and may transmit a stereoscopic image (e.g., an image of 360 degrees) and/or the haptic signal to the haptic device150through the communication interface.

According to various example embodiments, the first electronic device120may be a device coupled to the haptic device150. For example, the first electronic device120may be coupled to the housing-type haptic device150, via at least network160. The first electronic device120may output the stereoscopic image through a display, and the user may mount an HMD-type haptic device150on the user's head for example to admire the stereoscopic image output through the display of the first electronic device120. In this case, the first electronic device120may provide a user with various tangible effects such as vibration or the like, depending on the generated haptic signal.

According to various example embodiments, the first electronic device120may transmit information such that an image and/or a sound is capable of being output through the display and/or speakers of the haptic device150. For example, the first electronic device120may transmit the haptic signal and/or image information to the haptic device150. The haptic device150may output the haptic effect and/or may provide the user with an image and/or a sound, based on the signal or information received from the first electronic device120.

According to various example embodiments, the server130may perform at least part of operations performed by and/or attributed to the first electronic device120. For example, the server130may perform an operation such as stitching for generating an equi-rectangular projection (ERP) image, texture mapping, and/or the like. The server130may have a relatively high computing power and may reduce time required to process a stereoscopic image. The server130may transmit the processed stereoscopic image and/or haptic signal to the haptic device150.

The haptic device150may play the stereoscopic image (e.g., an image of 360 degrees). The haptic device150may include a display, one or more speakers capable of outputting a sound, one or more haptic modules capable of outputting a haptic effect, a communication interface (including communication interface circuitry) capable of receiving video data, audio data, motion data, and/or the like from an external device, a processor for computation, and the like.

The haptic device150may provide a user with various tangible effects based on the received haptic signal. For example, the haptic device150may be a wearable device such as an HMD, a wrist mounting device, or the like. For another example, the haptic device150may be an object, which may be disposed at a periphery of the user and which is contactable to the user's body, such as a moving chair or a vibration table.

The haptic device150may include one or more actuators and may drive the actuators based on at least the haptic signal that is received from the outside or is produced therein.

For example, the haptic device150may be the HMD and may include a module(s) that generates a haptic effect, such as a vibration, a wind, recoil, tilting, and/or the like, in/at the user's head part. For another example, the haptic device150may include a headphone-type module that generates the haptic effect (e.g., a tremor, vibration, or rotation effect) at a periphery of the user's head and/or ears.

For another example, the haptic device150may be a chair-type device that generates a haptic effect such as shake, vibration, tilting, rotation, or the like. For another example, the haptic device150may be in the form, which is worn on the body, such as a glove or suit, or may be in the form of a hand held controller.

The haptic device150may include various types of electronic devices that provide the tangible effect to the user. Hereinafter, the case where the haptic device150is an HMD will be described. However, embodiments of the present disclosure are not limited thereto.

According to various example embodiments, the network160may be a direct communication network (Bluetooth, Wi-Fi direct, or the like) between devices, or cellular communication network such as 3G, LTE, or the like.

FIG. 2illustrates various connection methods between a camera device and a haptic device, according to various example embodiments.

Referring toFIG. 2, in a connection type201, the camera device110and the haptic device150may be connected through the first electronic device120.

The first electronic device120may receive video data, audio data, or motion data, which is collected by the camera device110. The first electronic device120may produce a stereoscopic image composed of the video data and the audio data. In addition, the first electronic device120may generate haptic attribute information and a haptic signal based at least on the data received from the camera device110.

The first electronic device120may transmit the generated haptic signal to the haptic device150and may control the haptic device150to generate a haptic effect (e.g., vibration, movement, the generation of wind, or the like).

In a connection type202, the camera device110and the haptic device150may be connected through the network160. The network160may be short distance communication network such as direct connection between devices (Bluetooth or Wi-Fi direct) or the like, or may be medium/long distance communication network such as LTE communication, or the like. A 3-dimensional stereoscopic image may be made or a haptic signal may be generated, by one of the camera device110and/or the haptic device150. For example, the haptic device150may receive video data, audio data, or motion data, which is collected by the camera device110. The haptic device150may generate haptic attribute information and a haptic signal based on the received data.

In a connection type203, the camera device110and the haptic device150may be connected through the server130and/or network160. The server130may receive video data, audio data, or motion data, which is collected by the camera device110. The server130may produce a stereoscopic image composed of the video data and the audio data. In addition, the server130may generate haptic attribute information and a haptic signal based at least on the data received from the camera device110. The server130may transmit the stereoscopic image or the haptic signal to the haptic device150, optionally via network160.

In a connection type204, the camera device110may operate in conjunction with a first electronic device120/121. The haptic device150may operate in conjunction with a second electronic device122. The first electronic device121and the second electronic device122may be connected through the server130and/or network160.

The first electronic device121may receive video data, audio data, or motion data, which is collected by the camera device110. The first electronic device121may transmit the received data to the server130, optionally via network160. The server130may produce a stereoscopic image composed of the video data and the audio data. In addition, the server130may generate haptic attribute information and a haptic signal.

The second electronic device122may receive the stereoscopic image and the haptic signal from the server130, optionally via network160. The second electronic device122may transmit the received haptic signal to the haptic device150and may control the haptic device150to generate a haptic effect (e.g., vibration, movement, the generation of wind, or the like).

FIG. 3is a flowchart illustrating a haptic signal producing method, according to various example embodiments. The haptic signal producing method ofFIG. 3may be performed by one or more of the camera device110, the first electronic device120, the server130, and/or the haptic device150inFIG. 1.

Referring toFIG. 3, in operation310, the processor(s) (e.g., AP or MCU) of an electronic device (e.g., one or more of the camera device110, the first electronic device120, the server130, and/or the haptic device150inFIG. 1) may extract a first audio signal to be output through a speaker, based on audio data that is received from an external electronic device and/or stored in a memory. For example, the audio data may be data collected through the camera device110. For another example, the audio data may be data loaded from an internal memory, depending on a specified condition (e.g., relevance to video data).

The first audio signal may be a signal obtained by removing a noise signal, such as a wind noise, a background noise, and/or the like, from the audio data. The first audio signal may be a signal for outputting a sound played to a user through a speaker.

In operation320, the processor may extract a second audio signal obtained by at least removing the first audio signal from the audio data. The second audio signal may be a signal including at least part of a noise signal such as a wind noise, a background noise, and/or the like.

In operation330, the processor may generate a haptic signal based on the second audio signal. For example, the generated haptic signal may be used such that the haptic device150generates a haptic effect (e.g., vibration, shake, the generation of a wind, inertia effects, and/or the like) and provides the user with the haptic effect. Additional information about a method for generating the haptic signal based on the second audio signal will be given with reference toFIGS. 4 to 9.

FIG. 4illustrates an electronic device401, according to various embodiments. An electronic device401may be one of the camera device110, the first electronic device120, the server130, and the haptic device150inFIG. 1.

Referring toFIG. 4, the electronic device401may receive video data, audio data, and motion data from the camera device110. The video data, the audio data, and the motion data may be collected through the camera device110inFIG. 1.

The electronic device401may include an audio processor410, an audio rendering unit415, a video processor420, a video rendering unit425, a motion processor430, a haptic information generating unit440, and a haptic rendering unit445. The elements of the electronic device401may be separated depending upon function. However, embodiments of the present disclosure are not so limited. For example, the operations of the audio processor410, the video processor420, the motion processor430, and/or the haptic information generating unit440may be performed by one processor (e.g., AP or MCU) in an example embodiment. In another example, the operations of audio processor410, video processor420, and motion processor430may be performed by one processor. In another example, the operation of audio processor410and video processor420may be performed by a single processor. In another example, the operations of audio processor410and motion processor430may be performed by a single processor. Haptic rendering unit445and/or haptic information generator440may also be part of any of the aforesaid processors.

The audio processor410may receive and process the audio data. The audio data may be collected through the camera device110ofFIG. 1together with the video data or may be data recorded independently of the video data. The audio processor410may generate a first audio signal, which is to be transmitted to the audio rendering unit415through signal processing (e.g., noise cancellation, microphone array processing, gain control, DOA detection, compression, or the like) assigned to the audio data. The first audio signal may be a noise canceling signal obtained by removing noise data (e.g., a specific frequency component) from the input audio data. The first audio signal may be a signal for generating a sound output to the user's ears through a speaker.

According to various example embodiments, the audio processor410may extract a second audio signal (e.g., a noise, a wind noise, an ambient noise, or the like) obtained by removing the first audio signal (e.g., voice (a person's voice), a sound, the source of which is the closest to a microphone) from the input audio data. The audio processor410may transmit the second audio signal (audio feature) to the haptic information generating unit440. For example, the second audio signal may be a wind noise extracted from the audio data. Additional information about the second audio signal generated by the audio processor410may be given with reference toFIG. 5.

The audio rendering unit415may receive the first audio signal from the audio processor410. The audio rendering unit415may convert the first audio signal to a sound (e.g., 5.1 ch signal or 7.1 ch signal) that allows the user to feel the three-dimensional spatial sensation and may output the sound through the speaker.

The video processor420may process the input video data so as to output the processed data to a display. The video processor420may provide the processed data to the video rendering unit425. In various embodiments, the video processor420may extract feature information (visual feature) from the input video data and may transmit the extracted feature information to the haptic information generating unit440.

According to various example embodiments, the video processor420may transform a stereoscopic image (e.g., rotation transformation, distortion transformation, or the like) or may reduce the shake of the stereoscopic image, by reflecting information (e.g., camera pose, trajectory information, or the like) processed by the motion processor430.

The video rendering unit425may render a three-dimensional stereoscopic image based on the video data processed by the video processor420. For example, the video rendering unit425may form a virtual three-dimensional space associated with an area at which the user looks through a HMD, and a periphery of the corresponding area.

The motion processor430may calculate camera pose information, trajectory information, and/or motion event information based on the input motion data. For example, the motion processor430may calculate the camera pose or the translation (e.g., transposition and/or transformation) of the camera device110, based on the input motion data (e.g., data collected by GPS, gyroscopes, accelerometers, and/or magnetic field sensors).

The motion processor430may calculate trajectory information (or movement trajectory information) of the user based on translation information. The trajectory information may include information about a movement direction, a speed, or the like at a point of a specific time interval.

The motion processor430may transmit the camera pose information and/or the trajectory information to the haptic information generating unit440. The haptic information generating unit440may determine and/or change an attribute of a haptic event based on the transmitted information. For example, the haptic information generating unit440may determine an output direction of a haptic event (or a haptic effect) generating a wind, based on the trajectory information.

According to various example embodiments, the motion processor430may detect a motion event (e.g., a collision event, a shake, or the like) based on the motion data. The motion processor430may determine whether the motion event occurs, by calculating how much the motion data coincides with pre-stored motion pattern information. The haptic information generating unit440may determine or change the attribute of the haptic event based on the determined motion event.

The haptic information generating unit440may generate the haptic information based on the second audio signal and feature information of an image, camera pose information, trajectory information, or motion event information. The haptic information may include information about the time information (occurrence time, occurrence timing, duration, or the like), type (e.g., vibration, shake, or compression/decompression), intensity, direction, or the like of the haptic effect.

According to various example embodiments, the haptic information generating unit440may receive the audio data from the audio processor410and may reflect the audio data to the generation of the haptic information.

According to various example embodiments, the haptic information generating unit440may determine the attribute of the haptic information based on the second audio signal (e.g., a wind noise signal) received from the audio processor410. For example, the haptic information generating unit440may determine a type (e.g., a sound when the window of a vehicle is opened during driving, a breeze, a sound generated according to high-speed movement, a skiing sound, a sound generated when the user boards a ship, or the like) of the wind noise, based on the acoustic feature (spectral characteristic, cepstral characteristic, sound pressure level (SPL), periodicity, or the like) of the wind noise. The haptic information generating unit440may determine the attribute (e.g., haptic pattern) of the haptic event based on a type of the wind noise. Additional information about an operation of the haptic information generating unit440may be given with reference toFIG. 6.

The haptic rendering unit445may generate a haptic signal to be transmitted to the haptic device150, based on the haptic information generated by the haptic information generating unit440. The haptic rendering unit445may generate a haptic signal(s), which is to be played by haptic device(s), based on the attribute(s) of the haptic device(s) (e.g., a wind generator, a vibrator, and an inertial device) to be connected to the generated haptic information. For example, when generating the haptic signal, the haptic rendering unit445may use pose information of the haptic device150(in the case where there is direction information in the haptic event). Additional information about the haptic rendering unit445may be given with reference toFIG. 7.

FIG. 5is a flowchart illustrating an operation of an audio processor, according to various example embodiments.

Referring toFIG. 5, in operation510, the audio processor410may segment input audio data at specified intervals. Each of the segmented pieces of audio data (hereinafter referred to as an “audio frame”) may be at least partly overlapped in time. For example, the length of the audio frame may be 10 msec, and an overlapping section may be 5 msec.

In operation515, the audio processor410may perform Fourier transform on each audio frame to convert the audio frame to a frequency signal. Each audio frame may be expressed as magnitude information and phase information of the frequency signal.

In operation520, the audio processor410may extract background noise feature information based on a statistical feature of pieces of magnitude information corresponding to another audio frame different from the current audio frame. For example, the background noise feature information may be a signal to noise ratio (SNR) for each frequency.

In operation530, the audio processor410may determine a section, in which there is a wind noise, based on the statistical feature of pieces of magnitude information corresponding to another audio frame different from the current audio frame.

In operation535, the audio processor410may extract wind noise feature information about the section in which there is a wind noise. For example, the wind noise feature information may become the SNR for each frequency. Herein, noise N of the SNR may be the wind noise.

According to various example embodiments, the audio processor410may determine a frequency feature of the wind noise based on the wind noise feature information. For example, in the case where the wind noise feature information is the SNR for each frequency, the frequency feature of the wind noise may be determined by using the SNR for each frequency and the magnitude information.

According to various example embodiments, the audio processor410may generate a feature value such as cepstrum, a spectral slope, or the like through additional processing of the frequency feature of the wind noise. The audio processor410may determine a type of the wind noise based on the feature value.

According to various example embodiments, the background noise and the wind noise may be processed together without any distinction. The background noise and the wind noise may be processed as a single noise. The sound feature obtained by combining the background noise and the wind noise may be used to generate the haptic information. In this case, the audio processor410may generate a feature value such as Mel-frequency cepstral coefficients (MFCC), cepstrum, or the like with regard to a sound obtained by combining the background noise and the wind noise. The haptic information may be generated based on the feature value.

In operation540, the audio processor410may calculate the filter gain of a frequency area, to be used to remove a noise for example, based at least on the calculated background noise feature information and the calculated wind noise feature information.

In operation545, the audio processor410may apply the calculated filter gain to the audio data.

In operation550, the audio processor410may obtain first audio data, in which the noise is removed, by applying inverse Fourier transform or the like. The first audio signal may be obtained through overlap-add.

FIG. 6illustrates a configuration of a haptic information generating unit440, including haptic information generating circuitry, according to various example embodiments.

Referring toFIG. 6, the haptic information generating unit440may determine the attribute(s) of a haptic event based on a feature value(s) (e.g., acoustic feature) of a second audio signal (e.g., a noise signal). The haptic event may be a pre-defined signal pattern that allows the haptic effect to occur through the haptic device150. The haptic event may be defined or set in the various forms.

A second audio signal may be a signal obtained by removing a first audio signal (e.g., a person's voice, a sound having a specified pattern, a sound heard most loudly by a microphone, a sound analyzed as being generated closest to a microphone, or the like), which is to be output through a speaker, from audio data that is received from an external electronic device or is stored in a memory. The second audio signal may include a background sound, a wind sound, or the like that is classified as a noise signal.

Haptic attribute information may be information for setting a type, a duration (start time, end time), a persistence mode (intensity increase, intensity decrease), or the like of a haptic effect (e.g., an effect to generate vibration, inertia, shake, or the like).

According to various example embodiments, the haptic information generating unit440may include a feature calculating unit441and an attribute calculating unit442. If the second audio signal (e.g., a noise signal) is detected, the feature calculating unit441may extract an acoustic feature value of a noise.

In an example embodiment, the feature calculating unit441may segment the second audio signal at a preset time period to generate a plurality of audio frames. Each of a plurality of audio frames may have a section in which each of the plurality of audio frames overlaps another audio frame of a previous time period or a later time period. For example, the N-th audio frame may have a length of 10 ms, may have a section (the same data section) of 5 ms in which the N-th audio frame overlaps the (N−1)-th audio frame, and may have a section of 5 ms in which the N-th audio frame overlaps the (N+1)-th audio frame.

In an example embodiment, the feature calculating unit441may measure SPL associated with each of the audio frames. The attribute calculating unit442may determine the intensity of a haptic event corresponding to the measured SPL.

In an example embodiment, the feature calculating unit441may calculate a feature value (e.g., MFCC, a spectral slope, or the like) associated with each of a plurality of audio frames. For example, the feature calculating unit441may perform Fourier transform on a plurality of audio frames and may calculate a power spectrum by using a filter bank of Mel-scale. The feature calculating unit441may calculate a log value associated with the power spectrum of each Mel-scale, may perform discrete cosine transform, and may calculate the MFCC.

In an example embodiment, the feature calculating unit441may detect an interaural time difference (ITD) and/or an interaural level difference (ILD) of a second audio signal collected through a microphone array. The attribute calculating unit442may calculate the direction of a wind by using the detected ITD and/or ILD.

The attribute calculating unit442may determine a type of a noise based on the feature value calculated by the feature calculating unit441and may assign a haptic device type to be output based on the type of the noise.

For example, a type of a wind noise may include arbitrary various types of sounds such as a sound when the window of a vehicle is opened during driving, a breeze, a sound according to high-speed movement, and a sound caused by rapidly changing a movement direction, and the like. The sound when the window of the vehicle is opened during driving and the breeze may be assigned to the haptic device150including a wind generator, and the sound due to fast movement or the change of direction may be assigned to the haptic device150such as a Galvanic stimulator that generates an inertia effect.

The attribute calculating unit442may determine a type of a haptic event and/or a pattern of a haptic effect by using an extracted feature value. For example, if variance of the MFCC is greater than a preset value, the attribute calculating unit442may set a change value of a vibration according to the haptic effect to be relatively great. On the other hand, if the variance of the MFCC is less than a preset value, the attribute calculating unit442may set a change value of a vibration according to the haptic effect to be relatively small.

According to various example embodiments, the attribute calculating unit442may calculate direction information of an input microphone based on information of a motion sensor (e.g., GPS, acceleration sensor, a gyro sensor, a geomagnetic sensor, a magnetic sensor, or the like). For example, the attribute calculating unit442may determine the direction of a haptic event based on the calculated direction information (camera pose) of the microphone and the calculated direction of the wind noise.

FIG. 7illustrates a haptic rendering unit445, including haptic rendering circuitry, according to various example embodiments.

Referring toFIG. 7, the haptic rendering unit445may generate a haptic signal, which is to be played by one or more of the haptic device(s)150, based on the generated haptic attribute information. The haptic rendering unit445may generate a stereoscopic haptic signal for driving the haptic device150, based on the haptic attribute information. For example, in the case where a haptic effect output by the haptic device150is vibration, the haptic signal may include information about a drive start time, drive strength, a change in drive strength, a drive end time or the like of the vibrator. For another example, in the case where the haptic device150is a chair generating a shake, the haptic attribute information may include information about shake direction, shake strength, duration, or the like.

According to various example embodiments, the haptic information generated by the haptic information generating unit440may include information about a type of the haptic event, a haptic generating object, or the like, and may include information about a spatial location associated with the haptic event or the haptic generating object. The haptic attribute information may be input to the haptic rendering unit445.

The haptic rendering unit445may select the haptic event (or a haptic object) to be played or may change the attribute of the haptic event, based at least on the location information of the haptic device150(e.g., HMD).

For example, in the case where the location information of the HMD is in a state ((latitude, longitude, height, orientation θ, orientation φ)=(37°, 127°0′0.00000″, 0, 0°, 0°)), the intensity of the haptic event having haptic event coordinates (37°, 127°0′0.00001″, 0, 0°, 0°) may be smaller than the intensity of the haptic event having haptic event coordinates (37°, 127°0′0.00002″, 0, 0°, 0°).

For example, in the case where the location information of the HMD is in a state ((latitude, longitude, height, orientation θ, orientation φ)=(37°, 127°0′0.00000″, 0, 0°, 0°)), the haptic event of haptic event coordinates (37°, 127°0′0.00000″, 0, 0°, 0°) may generate a haptic effect on the right side of a user, and the haptic event of haptic event coordinate (37°, 127°0′0.00000″, 0, 90°, 0°) may generate the haptic effect in front of the user.

According to various embodiments, the haptic rendering unit445may include a priority determining unit446and a correction unit447.

In the case where the number of haptic events is plural, the priority determining unit446may assign the priority of each of the haptic events, based on the location information of the haptic device150(e.g., HMD).

For example, the priority determining unit446, including priority determining circuitry, may set a haptic event, which is closest to a point corresponding to the gaze direction (including a HMD pose) of the user (or wearer), from among a plurality of haptic events using the haptic device150to the highest priority. The correction unit447may set a weight associated with haptic strength, based on the set priority. The haptic event close to the point corresponding to the gaze direction may have a relatively high weight, and the haptic event far away from the point corresponding to the gaze direction may have a relatively low weight. The haptic attribute information may be combined and generated based on the weight.

The correction unit447, including correction circuitry, may finely adjust the haptic event. The correction unit447may correct the haptic attribute information based on the location information of the haptic device150(e.g., HMD) and/or the external information received from an external device. The external information may be provided online or offline through a web or various contents providers.

In an example embodiment, the external information may include weather information. The correction unit447may correct and/or add the haptic attribute information based on weather information. For example, in the case where it rains on the day of recording, the correction unit447may correct vibration haptic strength generated based on a sound to be relatively weak. In the case where the wind blows strongly on the day of recording, the correction unit447may correct wind haptic strength generated based on a sound to be relatively strong.

According to an embodiment, while collecting an image, a sound, or motion data by using a camera device, the external information may be generated based on the collected information at the same time. For example, the external information may be information collected at a point in time, when an image is captured, or information collected during operations such as editing, playback, and transmission after capturing the image.

FIG. 8Ais a block diagram of a camera device (110or801) generating haptic attribute information, according to various example embodiments. InFIG. 8A, an embodiment is exemplified as a configuration that generates a haptic signal based on an audio signal. However, embodiments of the present disclosure are not limited thereto. Although not illustrated inFIG. 8A, a camera device801(110or801) may additionally include a configuration such as a lens, an image sensor, an image processing unit, or the like.

Referring toFIG. 8A, the camera device801(or110) may capture an image through a camera and may generate a haptic signal using audio data recorded at the same time as the image capturing. The haptic signal may be directly transmitted to an external haptic device or may be transmitted to the external haptic device through another electronic device. The camera device801may include an audio processor810, a classification unit811, an audio rendering unit815, a haptic information generating unit840, and a haptic rendering unit845as a configuration processing the audio data. Some of all of these elements may be provided in one or more processors.

The audio processor810may receive the audio data collected through an embedded microphone of the camera device801or a separately mounted external microphone. The audio processor810may remove a noise signal from the audio data depending on a specified noise cancellation algorithm to extract a first audio signal to be transmitted to the audio rendering unit815.

The classification unit811may receive a second audio signal obtained by removing the first audio signal from the audio data, from the audio processor810. The classification unit811may classify the second audio signal depending on a specified tone and/or a frequency feature. For example, the classification unit811may separate a wind noise from a background noise.

The operations of the audio rendering unit815, the haptic information generating unit840, and the haptic rendering unit845may be the same as and similar to the operations of the audio rendering unit415, the haptic information generating unit440, and the haptic rendering unit445inFIG. 4and/orFIG. 5above, respectively.

FIG. 8Bis a schematic view illustrating generating a haptic signal through an external device separated from a camera device, according to various example embodiments.

Referring toFIG. 8B, a camera device801amay include the audio processor810as a configuration for processing audio data. The audio processor810may receive the audio data collected through an embedded microphone of the camera device801aor a separately mounted external microphone. The audio processor810may generate the first audio signal and the second audio signal. The audio processor810may remove the noise signal from the audio data depending on a specified noise cancellation algorithm to generate the first audio signal and may generate the second audio signal obtained by removing the first audio signal from the audio data.

The camera device801amay transmit the first audio signal and the second audio signal to an external device801cthrough a transmission network/device801b(e.g., a wired/wireless network).

For example, the external device801cmay be an electronic device (e.g., a smartphone) operating in conjunction with a haptic device (e.g., HMD), or may be, for another example, a server. The external device801cmay include a classification unit811a, an audio rendering unit815a, a haptic information generating unit840a, and a haptic rendering unit845aas a configuration for processing the audio data. The operations of the classification unit811a, the audio rendering unit815a, the haptic information generating unit840a, and the haptic rendering unit845amay be the same as and similar to the operations of the classification unit811, the audio rendering unit815, the haptic information generating unit840, and the haptic rendering unit845discussed above in connection withFIG. 8A, respectively.

FIG. 9Aillustrates a camera device901that generates a haptic signal by using sound collected through a microphone array, according to various example embodiments. InFIG. 9A, an embodiment is exemplified as a configuration that generates a haptic signal based on an audio signal. However, embodiments of the present disclosure are not limited thereto. Although not illustrated inFIG. 9A, a camera device901may additionally include a configuration such as a lens, an image sensor, an image processing unit, and/or the like.

Referring toFIG. 9A, the camera device901may generate a haptic signal based on audio data input by using a microphone array (a plurality of microphones).

In an example embodiment, the camera device901may include an audio processor910, an analysis unit911, a detection unit912, an audio rendering unit915, a haptic information generating unit940, and a haptic rendering unit945as a configuration for processing the audio data. All of these may be part of a single, or multiple, processor(s).

The audio processor910may receive audio data collected through the microphone array (the plurality of microphones). The audio data collected through the microphone array may have an audio feature more complex than audio data collected by using a single microphone. A distance between an object generating a sound and each of microphones may be different from each other, and the audio data collected through each of microphones may have a feature different from each other.

The audio processor910may remove a noise signal from the audio data depending on a specified noise cancellation algorithm to extract the first audio signal to be transmitted to the audio rendering unit915. The audio processor910may remove the first audio signal from the audio data to extract the second audio signal and may transmit the second audio signal to each of the analysis unit911and the detection unit912.

The analysis unit911may detect direction information (ITD, ILD, Generalized Sidelobe Canceller (GSC), Minimum Variance, or the like) of a sound from the second audio signal to transmit the direction information to the haptic information generating unit940. The detection unit912may detect a time section according to a preset pattern or frequency feature from the second audio signal to transmit information about the corresponding section to the haptic information generating unit940.

According to various example embodiments, the analysis unit911and the detection unit912may analyze a sound feature by using the audio data before the noise removal process or the audio data generated during the noise removal process.

The operations of the audio rendering unit915, the haptic information generating unit940, and the haptic rendering unit945may be the same as and similar to the operations of the audio rendering unit415, the haptic information generating unit440, and the haptic rendering unit445inFIG. 4, respectively.

FIG. 9Bis a schematic view for generating a haptic signal through an external device separated from a camera device, according to various example embodiments.

Referring toFIG. 9B, a camera device901amay include the audio processor910, the analysis unit911, the detection unit912, and the haptic information generating unit940as a configuration for processing the audio data. The operations of the audio processor910, the analysis unit911, the detection unit912, and the haptic information generating unit940may be the same as and similar to the operations of the corresponding configuration inFIG. 9A.

The audio processor910may remove a noise signal from audio data collected through a plurality of microphones depending on a specified noise cancellation algorithm to generate the first audio signal. The audio processor910may generate the second audio signal obtained by removing the first audio signal from the audio data. The haptic information generating unit940may generate haptic attribute information based on the second audio signal analyzed through the analysis unit911and the detection unit912.

The camera device901amay transmit the first audio signal and the haptic attribute information to an external device901cthrough a transmission network/device901b(e.g., a wired/wireless network).

For example, the external device901cmay be an electronic device (e.g., a smartphone) operating in conjunction with a haptic device (e.g., HMD), or may be, for another example, a server. The external device901cmay include an audio rendering unit915aand a haptic rendering unit945aas a configuration processing the audio data. The operations of the audio rendering unit915aand the haptic rendering unit945amay be the same as and similar to the operations of the audio rendering unit915and the haptic rendering unit945inFIG. 9A, respectively.

According to various example embodiments, a haptic signal producing method performed by an electronic device includes extracting a first audio signal to be output through a speaker, based on audio data received from an external electronic device or stored in the memory, generating a second audio signal obtained by removing the first audio signal from the audio data, and generating a haptic signal for outputting an haptic effect, based on the second audio signal.

According to various example embodiments, the extracting of the first audio signal includes removing a noise signal from a plurality of audio frames depending on a specified noise cancellation algorithm to extract the first audio signal.

According to various example embodiments, the generating of the haptic signal includes segmenting the second audio signal at a preset time period to generate a plurality of audio frames, and performing SPL on the plurality of audio frames to determine intensity of the haptic effect.

According to various example embodiments, the generating of the haptic signal includes calculating MFCC and/or a spectral slope of each of a plurality of audio frames.

According to various example embodiments, the generating of the haptic signal includes if variance of the MFCC is greater than a preset value, setting a change value of vibration according to the haptic effect to a specified value or more, and/or if variance of the MFCC is less than the preset value, setting the change value of vibration according to the haptic effect to the specified value or less.

According to various example embodiments, the generating of the second audio signal includes receiving additional information associated with the audio data from an external device, and correcting the haptic signal based on the additional information.

FIG. 10Aillustrates an external configuration of a haptic device, according to various example embodiments.

InFIG. 10A, a haptic device1001is an HMD. However, embodiments of the present disclosure are not limited thereto. For example, the haptic device1001may have a shape such as glasses.

Referring toFIG. 10A, the haptic device1001may be contacted or worn on/over both eyes of a user, and may be a device that displays an image. The haptic device1001may provide at least one of a see-through method providing augmented reality (AR) or a see-closed method providing virtual reality (VR).

The see-through method may provide additional information or an image in real time as one image while transmitting real external images to the user's eyes through a display.

The see-through method may provide visual effects or various sensory effects to the object or the virtual target or object while transmitting real external object(s) to the user's eyes through the display or a transparent/translucent lens. The haptic device1001may provide the user with additional information and/or an image about a real object through a see-through function. In an example embodiment, the haptic device1001may provide the user with additional information by using hologram or the like without the display or the lens.

The see-closed method may provide only contents provided through the display without passing through an external object, as an image.

According to various example embodiments, the haptic device1001may collect sensing data for recognizing a periphery of the haptic device1001or the user, from an embedded sensor. In addition, the haptic device1001may receive sensing data collected from an external source (e.g., an external device, an external server, or the like).

The haptic device1001may trace and interpret a wearer's physical head by using sensing data to set a relation between the wearer's physical body and a physical environment surrounding the wearer.

The haptic device1001may allow the wearer's TFOV to be classified into various areas, by tracing the wearer's physical head with regard to the wearer's body and an ambient environment. The area may be defined with regard to the body of the wearer of a HMD and an ambient environment. The haptic device1001may provide a system capable of arranging a virtual object or a virtual image with regard to the wearer's body and the ambient environment.

Generally, the virtual object may be disposed to be spaced apart from a first area or may be disposed in an unobtrusive area such that the virtual image is presented in the non-dangerous manner to reduce the visual occlusion of the first area. The virtual object may be disposed to be spaced apart from the first area to provide a context (e.g., to classify information as less relevance, less importance, or the like).

According to various example embodiments, the haptic device1001may include a main frame1010and a mounting part1020capable of fixing the main frame1010to the head of a user. The main frame1010may include an adjusting part1011for adjusting a display output. The mounting part1020may include a band formed of an elastic material and may allow the main frame1010to contact and/or be provided over eyes of the user's face.

The main frame1010may include at least one of a display or a transparent/translucent lens therein. In the case where the main frame1010includes the display, the display may be fixed to the main frame1010or may be removable. A lens assembly may be inserted between the display and the user's eye.

According to various example embodiments, the haptic device1001may include a wireless communication component providing a wireless communication function to communicate with a companion device (e.g., a smartphone), a server, or another network component. In an example embodiment, the haptic device1001may classify an area of TFOV of the wearer through a cloud-based service and may arrange the virtual object in the area. The haptic device1001may be configured to communicate with the server device directly or may communicate with the server device through the companion device (e.g., a smartphone or any other device) that is a local device compared with the wearer.

The haptic device1001may provide at least one of a see-through function providing AR or a see-closed function providing VR.

FIG. 10Billustrates an appearance of a removable type haptic device, according to various example embodiments.

Referring toFIG. 10B, a haptic device1002may include a main frame1030, a display device1032, a cover part1035, and a mounting part1040capable of fixing the main frame1030to a user's head. The main frame1030may include an adjusting part1031for adjusting a display output.

The main frame1030may include a space or a structure for containing the display device1032(e.g., LCD, OLED, LED, etc.). In addition, the main frame1030may further include a connector that is coupled to an electrical connection unit of a terminal and allows the main frame1030to communicate with the terminal.

According to various example embodiments, the main frame1030may include, for example, a touch panel on a part of the exterior thereof, as a user interface. The touch panel may include one or more display location adjusting parts or lens adjusting parts (not illustrated) on the external surface of the main frame1030.

For example, the main frame1030may include other type of a control device that controls the display device1032on the side surface thereof. For example, the control device may include at least one of a physical key, a physical button, a touch key, a joystick, a wheel key, a touch pad, or the like.

The touch pad may display a graphical user interface (GUI) capable of controlling a function of the terminal. (e.g., a GUI for controlling a sound or an image). The touch pad may receive a touch input (an input through a direct touch or a hovering input) of a user.

The main frame1030may be connected to the display device1032through an interface such as USB or the like, and the touch input that the touch panel receives may be transmitted to the display device1032. The display device1032may control a function corresponding to the touch input, in response to the touch input that the touch panel receives. For example, the display device1032may adjust a volume or may control the video playback, in response to the received touch input.

The main frame1030may include a face contact part (not illustrated) that contacts a user's face, inside the main frame1030. The face contact part may have a structure corresponding to curvature of the user's face and may at least partly include an elastic body. A part of the face contact part may include a nose recess having a shape into which the user's nose is inserted. A lens assembly that includes at least one lens may be inserted into a part of the face contact part, which is a location which the user's eyes face. When the user wears the haptic device1002, at least one surface of a lens may be exposed to the face contact part such that the user watches a screen of a display device with the user's eye.

The main frame1030may be implemented with a relatively lightweight material (e.g., plastic) for wearing comfort of the user. The main frame1030may include at least one of various other materials (e.g., glass, ceramic, metal (e.g., aluminum) or metal alloy (e.g., steel, stainless steel, titanium, or magnesium alloy)) for the strength and beauty.

One surface of the main frame1030may include a storage space in which the display device1032is capable of being mounted. A part forming the storage space may include the elastic material. The storage space may include a flexible material so as to change the size of the space, and may be equipped with the display device1032having various sizes.

The display device1032may be a device, which is used after being mounted in the haptic device1002or which independently operates after being separated from the haptic device1002. A lens assembly may be inserted between the display device1032and the user's eye.

FIG. 10Cis a block diagram illustrating an internal configuration of a haptic device, according to various example embodiments.FIG. 10Cis, but is not limited to, an example.

Referring toFIG. 10C, a haptic device1003may include micro controller unit (MCU)1050, a communication module1060including communication circuitry, a sensor module1070including sensor circuitry, an input device1080including input circuitry, a power management module1081, an eye tracker1091, a vibrator1092, and an adjustable optics1093(e.g., lens assembly). A part of elements of the haptic device1003may be included in a display device (e.g., a removable smartphone) in a main frame.

The MCU1050may include, for example, a processor and may control a plurality of hardware elements connected to the MCU1050by driving an operating system (OS) and/or an embedded software program.

The communication module1060may electrically connect the main frame of the haptic device1003to a smartphone by using wired and/or wireless communication and may transmit and receive data. The communication module1060may include a USB module, a Wi-Fi module, a BT module, a NFC module, or a GPS module. At least part (e.g., two or more) of the Wi-Fi module, the BT module, the GPS module, or the NFC module may be included in one integrated circuit (IC) or an IC package.

The sensor module1070may measure a physical quantity or may detect an operation state of the haptic device1003; the sensor module1070may convert the measured or detected information to an electrical signal. For example, the sensor module1070may include at least one of an acceleration sensor, a gyro sensor, a geomagnetic sensor, a magnetic sensor, a proximity sensor, a gesture sensor, a grip sensor, or a biometric sensor. The haptic device1003may detect the movement of a head of a user (wearer), by using the acceleration sensor, the gyro sensor, and the geomagnetic sensor. The haptic device1003may sense whether it is worn, by using the proximity sensor or the grip sensor. At least part of elements of the sensor module1070may be included in a removable smartphone.

The input device1080may include a touch pad and/or a button. The touch pad may recognize a touch input based on at least one of a capacitive detecting method, a resistive detecting method, an infrared detecting method, or an ultrasonic detecting method. Moreover, the touch pad may further include a control circuit. In the case of a capacitive detecting method, a physical contact or proximity recognition may be possible. The touch pad may further include a tactile layer. In this case, the touch pad may provide a tactile reaction to a user. The button may include, for example, a physical button, an optical key, or a keypad.

For example, the eye tracker1091may track the user's gaze by using at least one method of an electrical oculography (EOG) sensor, coil systems, dual purkinje systems, bright pupil systems, or dark pupil systems. In addition, the eye tracker1091may further include a micro camera for the eye tracking.

The adjustable optics1093may measure the inter-pupil distance (IPD) of a user and may adjust a distance of a lens and a location of the display of a smartphone such that the user admires an image suitable for his/her eyesight.

FIG. 11illustrates an electronic device in a network environment according to an example embodiment of the present disclosure.

An electronic device1101in a network environment1100according to various example embodiments of the present disclosure will be described with reference toFIG. 11. The electronic device1101may include a bus1110, a processor1120, a memory1130, an input/output interface1150including interface circuitry, a display1160, and a communication interface1170including communication interface circuitry. In various example embodiments of the present disclosure, at least one of the foregoing elements may be omitted or another element may be added to the electronic device1101.

The bus1110may include a circuit for connecting the above-mentioned elements1110to1170to each other and transferring communications (e.g., control messages and/or data) among the above-mentioned elements.

The processor1120may include at least one of a central processing unit (CPU), an application processor (AP), and/or a communication processor (CP). The processor1120may perform data processing or an operation related to communication and/or control of at least one of the other elements of the electronic device1101.

The memory1130may include a volatile memory and/or a nonvolatile memory. The memory1130may store instructions or data related to at least one of the other elements of the electronic device1101. According to an embodiment of the present disclosure, the memory1130may store software and/or a program1140. The program1140may include, for example, a kernel1141, middleware1143, an application programming interface (API)1145, and/or an application program (or an application)1147. At least a portion of the kernel1141, the middleware1143, and/or the API1145may be referred to as an operating system (OS).

The kernel1141may control or manage system resources (e.g., the bus1110, the processor1120, the memory1130, or the like) used to perform operations or functions of other programs (e.g., the middleware1143, the API1145, or the application program1147). Furthermore, the kernel1141may provide an interface for allowing the middleware1143, the API1145, or the application program1147to access individual elements of the electronic device1101in order to control or manage the system resources.

The middleware1143may serve as an intermediary so that the API1145or the application program1147communicates and exchanges data with the kernel1141.

Furthermore, the middleware1143may handle one or more task requests received from the application program1147according to a priority order. For example, the middleware1143may assign at least one application program1147a priority for using the system resources (e.g., the bus1110, the processor1120, the memory1130, or the like) of the electronic device1101. For example, the middleware1143may handle the one or more task requests according to the priority assigned to the at least one application, thereby performing scheduling or load balancing with respect to the one or more task requests.

The API1145, which is an interface for allowing the application1147to control a function provided by the kernel1141and/or the middleware1143, may include, for example, at least one interface or function (e.g., instructions) for file control, window control, image processing, character control, or the like.

The input/output interface1150may serve to transfer an instruction or data input from a user and/or another external device to (an)other element(s) of the electronic device1101. Furthermore, the input/output interface1150may output instructions or data received from (an)other element(s) of the electronic device1101to the user or another external device.

The display1160may include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display. The display1160may present various content (e.g., a text, an image, a video, an icon, a symbol, or the like) to the user. The display1160may include a touch screen, and may receive a touch, gesture, proximity or hovering input from an electronic pen or a part of a body of the user.

The communication interface1170may set communications between the electronic device1101and an external device (e.g., a first external electronic device1102, a second external electronic device1104, or a server1106). For example, the communication interface1170may be connected to a network1162via wireless communications or wired communications so as to communicate with the external device (e.g., the second external electronic device1104or the server1106).

The wireless communications may employ at least one of cellular communication protocols such as long-term evolution (LTE), LTE-advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), or global system for mobile communications (GSM). The wireless communications may include, for example, a short-range communications1164. The short-range communications may include at least one of wireless fidelity (Wi-Fi), Bluetooth, near field communication (NFC), magnetic stripe transmission (MST), or GNSS.

The MST may generate pulses according to transmission data and the pulses may generate electromagnetic signals. The electronic device1101may transmit the electromagnetic signals to a reader device such as a POS (point of sales) device. The POS device may detect the magnetic signals by using a MST reader and restore data by converting the detected electromagnetic signals into electrical signals.

The GNSS may include, for example, at least one of global positioning system (GPS), global navigation satellite system (GLONASS), BeiDou navigation satellite system (BeiDou), or Galileo, the European global satellite-based navigation system according to a use area or a bandwidth. Hereinafter, the term “GPS” and the term “GNSS” may be interchangeably used. The wired communications may include at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 832 (RS-232), plain old telephone service (POTS), or the like. The network1162may include at least one of telecommunications networks, for example, a computer network (e.g., local area network (LAN) or wide area network (WAN)), the Internet, or a telephone network.

The types of the first external electronic device1102and the second external electronic device1104may be the same as or different from the type of the electronic device1101. According to an example embodiment of the present disclosure, the server1106may include a group of one or more servers. A portion or all of operations performed in the electronic device1101may be performed in one or more other electronic devices (e.g., the first electronic device1102, the second external electronic device1104, and/or the server1106). When the electronic device1101should perform a certain function or service automatically or in response to a request, the electronic device1101may request at least a portion of functions related to the function or service from another device (e.g., the first electronic device1102, the second external electronic device1104, and/or the server1106) instead of or in addition to performing the function or service for itself. The other electronic device (e.g., the first electronic device1102, the second external electronic device1104, or the server1106) may perform the requested function or additional function, and may transfer a result of the performance to the electronic device1101. The electronic device1101may use a received result itself or additionally process the received result to provide the requested function or service. To this end, for example, a cloud computing technology, a distributed computing technology, or a client-server computing technology may be used.

FIG. 12is a block diagram illustrating an electronic device according to an example embodiment of the present disclosure.

Referring toFIG. 12, an electronic device1201may include, for example, a part or the entirety of the electronic device1101illustrated inFIG. 11. The electronic device1201may include at least one processor (e.g., AP)1210, a communication module1220including communication circuitry, a subscriber identification module (SIM)1224, a memory1230, a sensor module1240including sensing circuitry, an input device1250including input circuitry, a display1260, an interface1270including interface circuitry, an audio module1280, a camera module1291, a power management module1295, a battery1296, an indicator1297, and a motor1298.

The processor1210may run an operating system or an application program so as to control a plurality of hardware or software elements connected to the processor1210, and may process various data and perform operations. The processor1210may be implemented with, for example, a system on chip (SoC). According to an example embodiment of the present disclosure, the processor1210may further include a graphic processing unit (GPU) and/or an image signal processor. The processor1210may include at least a portion (e.g., a cellular module1221) of the elements illustrated inFIG. 12. The processor1210may load, on a volatile memory, an instruction or data received from at least one of other elements (e.g., a nonvolatile memory) to process the instruction or data, and may store various data in a nonvolatile memory.

The communication module1220may have a configuration that is the same as or similar to that of the communication interface1170ofFIG. 11. The communication module1220may include, for example, a cellular module1221, a Wi-Fi module1223, a Bluetooth (BT) module1225, a GNSS module1227(e.g., a GPS module, a GLONASS module, a BeiDou module, or a Galileo module), a NFC module1228, and a radio frequency (RF) module1229.

The cellular module1221may provide, for example, a voice call service, a video call service, a text message service, or an Internet service through a communication network. The cellular module1221may identify and authenticate the electronic device1201in the communication network using the subscriber identification module1224(e.g., a SIM card). The cellular module1221may perform at least a part of functions that may be provided by the processor1210. The cellular module1221may include a communication processor (CP).

Each of the Wi-Fi module1223, the Bluetooth module1225, the GNSS module1227and the NFC module1228may include, for example, a processor for processing data transmitted/received through the modules. According to some various example embodiments of the present disclosure, at least a part (e.g., two or more) of the cellular module1221, the Wi-Fi module1223, the Bluetooth module1225, the GNSS module1227, and the NFC module1228may be included in a single integrated chip (IC) or IC package.

The RF module1229may transmit/receive, for example, communication signals (e.g., RF signals). The RF module1229may include, for example, a transceiver, a power amp module (PAM), a frequency filter, a low noise amplifier (LNA), an antenna, or the like. According to another example embodiment of the present disclosure, at least one of the cellular module1221, the Wi-Fi module1223, the Bluetooth module1225, the GNSS module1227, or the NFC module1228may transmit/receive RF signals through a separate RF module.

The SIM1224may include, for example, an embedded SIM and/or a card containing the subscriber identity module, and may include unique identification information (e.g., an integrated circuit card identifier (ICCID)) or subscriber information (e.g., international mobile subscriber identity (IMSI)).

The memory1230(e.g., the memory1130) may include, for example, an internal memory1232or an external memory1234. The internal memory1232may include at least one of a volatile memory (e.g., a dynamic RAM (DRAM), a static RAM (SRAM), a synchronous dynamic RAM (SDRAM), or the like), a nonvolatile memory (e.g., a one-time programmable ROM (OTPROM), a programmable ROM (PROM), an erasable and programmable ROM (EPROM), an electrically erasable and programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory (e.g., a NAND flash memory, a NOR flash memory, or the like)), a hard drive, or a solid state drive (SSD).

The external memory1234may include a flash drive such as a compact flash (CF), a secure digital (SD), a Micro-SD, a Mini-SD, an extreme digital (xD), a MultiMediaCard (MMC), a memory stick, or the like. The external memory1234may be operatively and/or physically connected to the electronic device1201through various interfaces.

The sensor module1240may, for example, measure physical quantity or detect an operation state of the electronic device1201so as to convert measured or detected information into an electrical signal. The sensor module1240may include, for example, at least one of a gesture sensor1240A, a gyro sensor1240B, a barometric pressure sensor1240C, a magnetic sensor1240D, an acceleration sensor1240E, a grip sensor1240F, a proximity sensor1240G, a color sensor1240H (e.g., a red/green/blue (RGB) sensor), a biometric sensor1240I, a temperature/humidity sensor1240J, an illumination sensor1240K, or an ultraviolet (UV) sensor1240M. Additionally or alternatively, the sensor module1240may include, for example, an olfactory sensor (E-nose sensor), an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an iris recognition sensor, and/or a fingerprint sensor. The sensor module1240may further include a control circuit for controlling at least one sensor included therein. In some various embodiments of the present disclosure, the electronic device1201may further include a processor configured to control the sensor module1240as a part of the processor1210or separately, so that the sensor module1240is controlled while the processor1210is in a sleep state.

The input device1250may include, for example, a touch panel1252, a (digital) pen sensor1254, a key1256, or an ultrasonic input device1258. The touch panel1252may employ at least one of capacitive, resistive, infrared, and ultraviolet sensing methods. The touch panel1252may further include a control circuit. The touch panel1252may further include a tactile layer so as to provide a haptic feedback to a user.

The (digital) pen sensor1254may include, for example, a sheet for recognition which is a part of a touch panel or is separate. The key1256may include, for example, a physical button, an optical button, or a keypad. The ultrasonic input device1258may sense ultrasonic waves generated by an input tool through a microphone1288so as to identify data corresponding to the ultrasonic waves sensed.

The display1260(e.g., the display1160) may include a panel1262, a hologram device1264, or a projector1266. The panel1262may have a configuration that is the same as or similar to that of the display1160ofFIG. 11. The panel1262may be, for example, flexible, transparent, or wearable. The panel1262and the touch panel1252may be integrated into a single module. The hologram device1264may display a stereoscopic image in a space using a light interference phenomenon. The projector1266may project light onto a screen so as to display an image. The screen may be disposed in the inside or the outside of the electronic device1201. According to an embodiment of the present disclosure, the display1260may further include a control circuit for controlling the panel1262, the hologram device1264, or the projector1266.

The interface1270may include, for example, an HDMI1272, a USB1274, an optical interface1276, or a D-subminiature (D-sub)1278. The interface1270, for example, may be included in the communication interface1170illustrated inFIG. 11. Additionally or alternatively, the interface1270may include, for example, a mobile high-definition link (MHL) interface, an SD card/multi-media card (MMC) interface, or an infrared data association (IrDA) interface.

The audio module1280may convert, for example, a sound into an electrical signal or vice versa. At least a portion of elements of the audio module1280may be included in the input/output interface1150illustrated inFIG. 11. The audio module1280may process sound information input or output through a speaker1282, a receiver1284, an earphone1286, or the microphone1288.

The camera module1291may be, for example, a device for shooting a still image or a video. According to an example embodiment of the present disclosure, the camera module1291may include at least one image sensor (e.g., a front sensor or a rear sensor), a lens, an image signal processor (ISP), or a flash (e.g., an LED or a xenon lamp).

The power management module1295may manage power of the electronic device1201. According to an example embodiment of the present disclosure, the power management module1295may include a power management integrated circuit (PMIC), a charger integrated circuit (IC), or a battery or gauge. The PMIC may employ a wired and/or wireless charging method. The wireless charging method may include, for example, a magnetic resonance method, a magnetic induction method, an electromagnetic method, or the like. An additional circuit for wireless charging, such as a coil loop, a resonant circuit, a rectifier, or the like, may be further included. The battery gauge may measure, for example, a remaining capacity of the battery1296and a voltage, current or temperature thereof while the battery is charged. The battery1296may include, for example, a rechargeable battery and/or a solar battery.

The indicator1297may display a specific state of the electronic device1201or a part thereof (e.g., the processor1210), such as a booting state, a message state, a charging state, or the like. The motor1298may convert an electrical signal into a mechanical vibration, and may generate a vibration or haptic effect. Although not illustrated, a processing device (e.g., a GPU) for supporting a mobile TV may be included in the electronic device1201. The processing device for supporting a mobile TV may process media data according to the standards of digital multimedia broadcasting (DMB), digital video broadcasting (DVB), MediaFLO™, or the like.

Each of the elements described herein may be configured with one or more components, and the names of the elements may be changed according to the type of an electronic device. In various embodiments of the present disclosure, an electronic device may include at least one of the elements described herein, and some elements may be omitted or other additional elements may be added. Furthermore, some of the elements of the electronic device may be combined with each other so as to form one entity, so that the functions of the elements may be performed in the same manner as before the combination.

According to various example embodiments, an electronic device includes a memory, a communication interface, and a processor electrically connected to the memory and the communication interface, wherein the processor is configured to extract a first audio signal to be output through a speaker, based on audio data received from an external electronic device or stored in the memory, generate a second audio signal obtained by removing the first audio signal from the audio data, and generate a haptic signal for outputting an haptic effect, based on the second audio signal.

According to various embodiments, the electronic device is one of a camera device, a haptic device, or a server device or is a device interlocking to control at least one of the camera device or the haptic device.

According to various embodiments, the processor is configured to segment the audio data at a preset time period to generate a plurality of audio frames, and perform Fourier transform on the plurality of audio frames.

According to various embodiments, the audio frame has a section in which the audio frame overlaps another audio frame of a time period before or after the audio frame.

According to various embodiments, the processor is configured to remove a noise signal from the plurality of audio frames depending on a specified noise cancellation algorithm to extract the first audio signal.

According to various embodiments, the second audio signal is a signal to noise ratio (SNR) for each frequency included in the plurality of audio frames.

According to various embodiments, the processor is configured to segment the second audio signal at a preset time period to generate a plurality of audio frames, and perform sound pressure level (SPL) on the plurality of audio frames to determine intensity of the haptic effect.

According to various embodiments, the processor is configured to analyze a statistical feature of the plurality of audio frames, and determine a type or an output method of the haptic effect.

According to various embodiments, the processor is configured to calculate Mel-frequency cepstral coefficients (MFCC) or a spectral slope of each of the plurality of audio frames.

According to various embodiments, the processor is configured to if variance of the MFCC is greater than a preset value, set a change value of vibration according to the haptic effect to a specified value or more.

According to various embodiments, the processor is configured to if variance of the MFCC is less than a preset value, set a change value of vibration according to the haptic effect to a specified value or less.

According to various embodiments, audio data is collected through a microphone array, and the processor is configured to detect an interaural time difference (ITD) and/or an interaural level difference (ILD) of the second audio signal.

According to various embodiments, the processor is configured to determine a direction of the haptic effect based on the detected ITD and/or ILD.

According to various embodiments, the processor is configured to receive additional information associated with the audio data from an external device, and correct the haptic signal based on the additional information.

FIG. 13is a block diagram illustrating a program module according to an example embodiment of the present disclosure.

Referring toFIG. 13, a program module1310(e.g., the program1140) may include an operating system (OS) for controlling a resource related to an electronic device (e.g., the electronic device1101) and/or various applications (e.g., the application program1147) running on the OS. The operating system may be, for example, Android, iOS, Windows, Symbian, Tizen, or the like.

The program module1310may include a kernel1320, a middleware1330, an API1360, and/or an application1370. At least a part of the program module1310may be preloaded on an electronic device or may be downloaded from an external electronic device (e.g., the first electronic device1102, the second external electronic device1104, or the server1106).

The kernel1320(e.g., the kernel1141) may include, for example, a system resource manager1321or a device driver1323. The system resource manager1321may perform control, allocation, or retrieval of a system resource. According to an embodiment of the present disclosure, the system resource manager1321may include a process management unit, a memory management unit, a file system management unit, or the like. The device driver1323may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a Wi-Fi driver, an audio driver, or an inter-process communication (IPC) driver.

The middleware1330, for example, may provide a function that the applications1370require in common, or may provide various functions to the applications1370through the API1360so that the applications1370may efficiently use limited system resources in the electronic device. According to an embodiment of the present disclosure, the middleware1330(e.g., the middleware1143) may include at least one of a runtime library1335, an application manager1341, a window manager1342, a multimedia manager1343, a resource manager1344, a power manager1345, a database manager1346, a package manager1347, a connectivity manager1348, a notification manager1349, a location manager1350, a graphic manager1351, and a security manager1352.

The runtime library1335may include, for example, a library module that a complier uses to add a new function through a programming language while the application1370is running. The runtime library1335may perform a function for input/output management, memory management, or an arithmetic function.

The application manager1341may mange, for example, a life cycle of at least one of the applications1370. The window manager1342may manage a GUI resource used in a screen. The multimedia manager1343may recognize a format required for playing various media files and may encode or decode a media file using a codec matched to the format. The resource manager1344may manage a resource such as a source code, a memory, or a storage space of at least one of the applications1370.

The power manager1345, for example, may operate together with a basic input/output system (BIOS) to manage a battery or power and may provide power information required for operating the electronic device. The database manager1346may generate, search, or modify a database to be used in at least one of the applications1370. The package manager1347may manage installation or update of an application distributed in a package file format.

The connectivity manger1348may manage wireless connection of Wi-Fi, Bluetooth, or the like. The notification manager1349may display or notify an event such as message arrival, appointments, and proximity alerts in such a manner as not to disturb a user. The location manager1350may manage location information of the electronic device. The graphic manager1351may manage a graphic effect to be provided to a user or a user interface related thereto. The security manager1352may provide various security functions required for system security or user authentication. According to an embodiment of the present disclosure, in the case in which an electronic device (e.g., the electronic device1101) includes a phone function, the middleware1330may further include a telephony manager for managing a voice or video call function of the electronic device.

The middleware1330may include a middleware module for forming a combination of various functions of the above-mentioned elements. The middleware1330may provide a module specialized for each type of an operating system to provide differentiated functions. Furthermore, the middleware1330may delete a part of existing elements or may add new elements dynamically.

The API1360(e.g., the API1145) which is, for example, a set of API programming functions may be provided in different configurations according to an operating system. For example, in the case of Android or iOS, one API set may be provided for each platform, and, in the case of Tizen, at least two API sets may be provided for each platform.

The application1370(e.g., the application program1147), for example, may include at least one application capable of performing functions such as a home1371, a dialer1372, an SMS/MMS1373, an instant message (IM)1374, a browser1375, a camera1376, an alarm1377, a contact1378, a voice dial1379, an e-mail1380, a calendar1381, a media player1382, an album1383, a clock1384, health care (e.g., measure an exercise amount or blood sugar), or environmental information provision (e.g., provide air pressure, humidity, or temperature information).

According to an example embodiment of the present disclosure, the application1370may include an information exchange application for supporting information exchange between the electronic device (e.g., the electronic device1101) and an external electronic device (e.g., the first electronic device1102or the second external electronic device1104). The information exchange application may include, for example, a notification relay application for relaying specific information to the external electronic device or a device management application for managing the external electronic device.

For example, the notification relay application may have a function for relaying, to an external electronic device (e.g., the first electronic device1102or the second external electronic device1104), notification information generated in another application (e.g., an SMS/MMS application, an e-mail application, a health care application, an environmental information application, or the like) of the electronic device. Furthermore, the notification relay application may receive notification information from the external electronic device and may provide the received notification information to the user.

The device management application, for example, may manage (e.g., install, delete, or update) at least one function (e.g., turn-on/turn off of the external electronic device itself (or some elements) or the brightness (or resolution) adjustment of a display) of the external electronic device (e.g., the first electronic device1102or the second external electronic device1104) communicating with the electronic device, an application running in the external electronic device, or a service (e.g., a call service, a message service, or the like) provided from the external electronic device.

According to an example embodiment of the present disclosure, the application1370may include a specified application (e.g., a healthcare application of a mobile medical device) according to an attribute of the external electronic device (e.g., the first electronic device1102or the second external electronic device1104). The application1370may include an application received from an external electronic device (e.g., the first electronic device1102or the second external electronic device1104). The application1370may include a preloaded application or a third-party application downloadable from a server. The names of the elements of the program module1310illustrated may vary with the type of an operating system.

According to various example embodiments of the present disclosure, at least a part of the program module1310may be implemented with software, firmware, hardware, or a combination thereof. At least a part of the program module1310, for example, may be implemented (e.g., executed) by a processor (e.g., the processor1210). At least a part of the program module1310may include, for example, a module, a program, a routine, sets of instructions, or a process for performing at least one function.

At least a part of devices (e.g., modules or functions thereof) or methods (e.g., operations) according to various embodiments of the present disclosure may be implemented as instructions stored in a computer-readable storage medium in the form of a program module. In the case where the instructions are performed by a processor (e.g., the processor1120), the processor may perform functions corresponding to the instructions. The computer-readable storage medium may be, for example, the memory1130.

A computer-readable recording medium may include a hard disk, a floppy disk, a magnetic medium (e.g., a magnetic tape), an optical medium (e.g., CD-ROM, digital versatile disc (DVD)), a magneto-optical medium (e.g., a floptical disk), or a hardware device (e.g., a ROM, a RAM, a flash memory, or the like). The program instructions may include machine language codes generated by compilers and high-level language codes that can be executed by computers using interpreters. The above-mentioned hardware device may be configured to be operated as one or more software modules for performing operations of various embodiments of the present disclosure and vice versa.

A module or a program module according to various embodiments of the present disclosure may include at least one of the above-mentioned elements, or some elements may be omitted or other additional elements may be added. Operations performed by the module, the program module or other elements according to various embodiments of the present disclosure may be performed in a sequential, parallel, iterative or heuristic way. Furthermore, some operations may be performed in another order or may be omitted, or other operations may be added.