Method for providing information mapped between a plurality of inputs and electronic device for supporting the same

An electronic device is provided. The electronic device includes a microphone, a touch screen display, a processor, and memory. The memory stores instructions, when executed, causing the at least one processor to: output a screen where a specified application is executed, the screen including a first interface configured to support recording a voice input through the at least one microphone and a second interface configured to receive content input of a user, on the touch screen display; pairing and storing in the memory each one of a plurality of recorded voice inputs to the first interface with a corresponding one of a plurality of content inputs to the second interface, thereby forming a voice-content pair, wherein the voice input and content input of each voice-content pair are contemporaneously recorded and received; receive a user input based on a keyword for specified content; select at least one recorded voice-content pair in response to the user input; convert recorded voice inputs of the at least one voice-content pair into text data; and output the converted text data on the touch screen display.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. § 119 to Korean Patent Application No. 10-2018-0032444, filed on Mar. 21, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated by reference herein its entirety.

BACKGROUND

The disclosure relates to technologies of providing information mapped between a plurality of inputs.

2. Description of Related Art

In recent years, electronic devices have supported a variety of input modes as a way of aiming at interaction with users. For example, an electronic device may support a voice-based input mode according to user utterances as well as an input mode using a software input panel (SIP) keyboard, a stylus pen, or the like. Alternatively, the electronic device may support a multimodal interface input mode for operating the above-mentioned input modes in a complex manner.

SUMMARY

In connection with recording information associated with a specific event, an electronic device may receive a multimodal interface input from a user. For example, the electronic device may receive and record a voice input associated with the event and may simultaneously receive and store a content input (e.g., a text input or the like) including information associated with the voice. In this case, the electronic device may separately process recording data and content data, which are caused by the same event. Thus, the electronic device may fail to satisfy user needs to identify information (e.g., recording information and content information) about the same event in a complex manner or may fail to provide convenience.

Aspects of the present disclosure are to address at least the above-mentioned problems and/or disadvantages and to provide at least the advantages described below. Accordingly, an aspect of the present disclosure is to provide a method for providing information mapped between a plurality of inputs to map and provide data according to a multimodal interface input in a linked manner depending on a specified criterion and an electronic device for supporting the same.

In accordance with an aspect of the present disclosure, an electronic device is provided. An electronic device may include at least one microphone, a touch screen display, at least one processor configured to be electrically connected with the at least one microphone and the touch screen display, and memory configured to be electrically connected with the at least one processor.

According to an embodiment, the memory stores instructions, when executed, causing the at least one processor to: output a screen where a specified application is executed, the screen including a first interface configured to support recording a voice input through the at least one microphone and a second interface configured to receive content input of a user, on the touch screen display; pairing and storing in the memory each one of a plurality of recorded voice inputs to the first interface with a corresponding one of a plurality of content inputs to the second interface, thereby forming a voice-content pair, wherein the voice input and content input of each voice-content pair are contemporaneously recorded and received; receive a user input based on a keyword for specified content; select at least one recorded voice-content pair in response to the user input; convert recorded voice inputs of the at least one voice-content pair into text data; and output the converted text data on the touch screen display.

In accordance with another aspect of the present disclosure, a method for providing information mapped between a plurality of inputs in an electronic device is provided. The method includes outputting a first screen where a specified application is executed, the first screen including a first interface for supporting to record a voice input to the electronic device and a second interface for supporting a content input of a user; pair and store in the memory each one of a plurality of recorded voice inputs to the first interface with a corresponding one of a plurality of content inputs to the second interface, thereby forming a voice-content pair, wherein the voice input and content input of each voice-content pair are contemporaneously recorded and received; receiving a user input based on a keyword for specified content; select at least one recorded voice-content pair in response to the user input; converting recorded voice inputs of the at least one voice-content pair into text data; and outputting the converted text data.

DETAILED DESCRIPTION

Hereinafter, certain embodiments of the disclosure may be described with reference to accompanying drawings. Accordingly, those of ordinary skill in the art will recognize that modification, equivalent, and/or alternative on the certain embodiments described herein can be variously made without departing from the scope and spirit of the disclosure. With regard to description of drawings, similar components may be marked by similar reference numerals.

The terms, such as “first”, “second”, and the like used in the disclosure may be used to refer to various components regardless of the order and/or the priority and to distinguish the relevant components from other components, but do not limit the components. For example, “a first user device” and “a second user device” indicate different user devices regardless of the order or priority. For example, without departing the scope of the disclosure, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

It will be understood that when an component (e.g., a first component) is referred to as being “(operatively or communicatively) coupled with/to” or “connected to” another component (e.g., a second component), it may be directly coupled with/to or connected to the other component or an intervening component (e.g., a third component) may be present. In contrast, when an component (e.g., a first component) is referred to as being “directly coupled with/to” or “directly connected to” another component (e.g., a second component), it should be understood that there are no intervening component (e.g., a third component).

An electronic device according to certain embodiments of the disclosure may include at least one of, for example, smartphones, tablet personal computers (PCs), mobile phones, video telephones, electronic book readers, desktop PCs, laptop PCs, netbook computers, workstations, servers, personal digital assistants (PDAs), portable multimedia players (PMPs), Motion Picture Experts Group (MPEG-1 or MPEG-2) Audio Layer 3 (MP3) players, mobile medical devices, cameras, or wearable devices. According to certain embodiments, the wearable device may include at least one of an accessory type (e.g., watches, rings, bracelets, anklets, necklaces, glasses, contact lens, or head-mounted-devices (HMDs), a fabric or garment-integrated type (e.g., an electronic apparel), a body-attached type (e.g., a skin pad or tattoos), or a bio-implantable type (e.g., an implantable circuit).

According to another embodiment, an electronic device may include at least one of various medical devices (e.g., various portable medical measurement devices (e.g., a blood glucose monitoring device, a heartbeat measuring device, a blood pressure measuring device, a body temperature measuring device, and the like), a magnetic resonance angiography (MRA), a magnetic resonance imaging (MRI), a computed tomography (CT), scanners, and ultrasonic devices), navigation devices, Global Navigation Satellite System (GNSS), event data recorders (EDRs), flight data recorders (FDRs), vehicle infotainment devices, electronic equipment for vessels (e.g., navigation systems and gyrocompasses), avionics, security devices, head units for vehicles, industrial or home robots, automated teller machines (ATMs), points of sales (POSs) of stores, or internet of things (e.g., light bulbs, various sensors, electric or gas meters, sprinkler devices, fire alarms, thermostats, street lamps, toasters, exercise equipment, hot water tanks, heaters, boilers, and the like).

According to an embodiment, the electronic device may include at least one of parts of furniture or buildings/structures, electronic boards, electronic signature receiving devices, projectors, or various measuring instruments (e.g., water meters, electricity meters, gas meters, or wave meters, and the like). According to certain embodiments, the electronic device may be one of the above-described devices or a combination thereof Δn electronic device according to an embodiment may be a flexible electronic device. Furthermore, an electronic device according to an embodiment of the disclosure may not be limited to the above-described electronic devices and may include other electronic devices and new electronic devices according to the development of technologies.

Hereinafter, electronic devices according to certain embodiments will be described with reference to the accompanying drawings. In the disclosure, the term “user” may refer to a person who uses an electronic device or may refer to a device (e.g., an artificial intelligence electronic device) that uses the electronic device.

FIG. 1is a drawing illustrating an operation of an electronic device according to an embodiment. In certain embodiments the user can record voice inputs in a first interface10and provide content input in a second interface20. In one embodiment (left side), the content input can be entered using a keyboard30. In another embodiment (right side) the content input can be entered using a stylus40on a writing interface20.

Referring toFIG. 1, an electronic device100may execute an application (e.g., a note application). The application can supports user handwriting input based on a specified input interface in response to user control and may output a screen133where the application is executed. In an embodiment, the input interface may include at least one of a software input panel (SIP) keyboard30output on one region of a screen of the electronic device100or a stylus pen40mounted on the electronic device100. In this regard, the at least one menu tab134and/or136for specifying an input interface which supports the user handwriting input (or setting a user handwriting input mode) may be provided on the screen133where the application is executed. For example, when a user input (e.g., touch) is applied to the first menu tab134, the electronic device100may output the SIP keyboard30. A user may provide a text input using the SIP keyboard30. When a user input (e.g., touch) is applied to the second menu tab136, the electronic device100may detect a signal of the stylus pen40. The user may provide a handwriting input using the stylus pen40. According to the above-mentioned details, the user handwriting input may be understood as the meaning including a text input or a marking input (e.g., an ungrammatical asterisk, underscore, tilde, check, or the like) based on the SIP keyboard30or the stylus pen40. Hereinafter, the user handwriting input may be called a content input.

In an embodiment, the screen133where the application is executed may include a first interface10for operating a recording function loaded into the electronic device100. The electronic device100may record and store a voice which occurs in an environment where the electronic device100is operated (or a voice received by the electronic device100), in response to user control for the first interface10, or may output the stored recording data. Furthermore, the screen133where the application is executed may include a second interface20for entering any information, such as content input. The electronic device100may receive a content input of the user, associated with entering the information, and may display the received content input on at least one region of the second interface20. According to an embodiment, the second interface20may be expanded in vertical or horizontal scrolling form according to an amount, size, location, or the like of the input content.

In an embodiment, concurrently with recording a voice input received in response to user control for the first interface10, the electronic device100may receive a content input of the user and may display the received content input on the second interface20. For example, the electronic device100may receive a voice input and a content input, which are associated with the same event (e.g., a lecture or the like), and may record the received voice input or may display the received content input on its screen. In this operation, the electronic device100may map and store the recorded data and the input content data on the basis of the same time. For example, the electronic device100may map data recorded at the same time to content data input at the same time to store the mapped data in a database or to store the mapped data in the form of text.

In certain embodiments, mapping can include creating a data structure which associates the content input with the voice input. The electronic device100may store a plurality of recorded voice inputs and a corresponding plurality of content inputs. Each one of the plurality of recorded voice inputs can be mapped with a corresponding one of the plurality of content inputs, thereby forming voice-content pairs, wherein the voice input and the content input of the voice-content pairs are recorded and input contemporaneously, or during a common time period. The voice-content pairs can be indexed by time period and have various hashtag keywords.

In an embodiment, the electronic device100may visually or audibly provide second data mapped to first data requested or selected from a user input, in response to the user input provided to the first interface10or the second interface20. For example, when data recorded at a first time is requested from user control for the first interface10, concurrently with responding to the recording data request at the first time, the electronic device100may visually specify content data mapped to the recording data at the first time (or input at the first time) with reference to the database or index. Similarly, when content input at a second time is selected from user control for the second interface20, concurrently with responding to the content selection at the second time, the electronic device100may visually or audibly provide recording data mapped to content data at the second time (recorded at the second time). Hereinafter, a description will be given of certain embodiments for providing information in response to a user input based on mapping of data (e.g., recording data and content data) generated or input at the same time and function operations of the components of the electronic device100implementing the embodiments.

FIG. 2is a block diagram illustrating a configuration of an electronic device according to an embodiment. The electronic device100communicates with an external device200over a network300, such as, but not limited to, the internet

Referring toFIG. 2, an electronic device100may include at least one of a communication circuitry110, an input/output device120, a memory130, a display140, or one or more processors150(from hereinafter, “processor” shall be understood to mean one or more processors regardless of the singular context). According to certain embodiments, the electronic device100may fail to include at least one of the above-mentioned components or may further include another component. For example, the electronic device100may further include a power supply (e.g., a battery) for supplying power to the components, a camera device (e.g., a front camera and/or a rear camera) for capturing an image (e.g., a still image, a video, or the like) for an area around the electronic device100, or a sensor device (e.g., a proximity sensor, an illumination sensor, an acceleration sensor, a fingerprint sensor, an iris sensor, or the like) for sensing a variety of information about an environment where the electronic device100is operated. Alternatively, the electronic device100may further include components of an electronic device901which will be described below with reference toFIG. 9. At least some of function operations of the components of the electronic device100, which will be described below, may be understood as being implemented under control of the processor150based on at least one instruction stored in the memory130.

The communication circuitry110may support communication between the electronic device100and at least one external device200. For example, the communication circuitry110may establish a connection with network300according to a defined protocol with the at least one external device200and may access the network300based on wired communication or wireless communication to transmit a signal or data to the at least one external device200or receive a signal or data from the at least one external device200. According to certain embodiments, the at least one external device200may include the same type of device (e.g., a mobile device or the like) as the electronic device100or a different type of device (e.g., a server or the like) from the electronic device100.

The input/output device120may receive or support the above-mentioned voice input or content input. For example, the input/output device120may include at least one microphone, an SIP keyboard (e.g., an SIP board30ofFIG. 1), a keyboard, a mouse, a touch pad, a stylus pen (e.g., a stylus pen40ofFIG. 1), or the like, which can be mounted on one region of the electronic device100, is output on the electronic device100, or is connected to the electronic device100, and may receive a signal or data according to the voice input or the content input or may deliver the received signal or data to another component of the electronic device100. Alternatively, the input/output device120may include at least one speaker loaded into one region of the electronic device100and may output a voice signal or data, which is generated by the electronic device100or is received from the at least one external device200, to the outside of the electronic device100. In an embodiment, at least a portion of the input/output device120(e.g., at least one microphone or the like) may always be controlled to be active (e.g., always on). Alternatively, when a user input (e.g., touch) is applied to the above-mentioned first interface (e.g., a first interface ofFIG. 1), the at least portion of the input/output device120may be triggered and enabled. In an embodiment, the input/output device120may include a recording module for recording a voice signal or data input via the microphone.

The memory130may store at least one signal or data associated with operating the electronic device100or may store at least one instruction associated with function operations of the components of the electronic device100. For example, the memory130may store voice data input based on the input/output device120, content data, or recording data for the voice. In an embodiment, in connection with storing the recording data or the content data, the memory130may store the recording data and the content data at a specific time, which are mapped under control of the processor150, in the form of a database or index. In this operation, the processor150may include information about the specific time, as identification information about the mapped recording data and content data at the specific time, in the database or index. In an embodiment, the memory130may store at least one application131, which is loaded in the preloaded form when the electronic device100is manufactured or is downloaded in the third-party form from an online market (e.g., App Store). According to an embodiment, the at least one application131may include an application (e.g., a node application) which supports at least one of controlling a function operation of the input/output device120(e.g., the recording module or the like) (e.g., recording input voice data) or entering information (e.g., a content input) of a user.

As used herein, “memory” refers to the entire memory system of an electronic device as opposed to a particular module, chip, circuit, or type. Memory shall refer to both instruction memory and data memory.

The display140may output various screens. For example, the display140may output a screen when the electronic device100is operated (e.g., a home screen, a screen where the application131is executed (e.g., a screen133where an application is executed inFIG. 1)) in response to user control or specified scheduling information. Alternatively, the display140may display text or marking (e.g., an ungrammatical asterisk, underscore, tilde, check, or the like) according to a user input (e.g., a content input) on the screen where the electronic device100is operated. In certain embodiments, the display140may be implemented as a touch screen display including a display panel, a cover glass, and a touch panel (or a touch sensor). The display panel may output a related screen based on a specified driving signal. In this regard, the display140may include a display driver circuit for delivering a driving signal to the display panel. The display driver circuit may be electrically connected with the display panel and may deliver a driving signal corresponding to content information received from the processor150to the display panel at a specified frame rate. The cover glass may be disposed on an upper portion of the display panel to transmit light as the display panel outputs a screen. Furthermore, a user input (e.g., touch, drag, press, hovering, or the like) by a user's body (e.g., a finger) or a stylus pen (e.g., a stylus pen40ofFIG. 1) may be applied to at least one region of the cover glass. The touch panel may perform detection (e.g., capacitive detection, resistive detection, infrared detection, ultrasonic detection, or the like) of a signal according to a user input (e.g., touch, drag, press, hovering, or the like) to output an electrical signal and may deliver information about the electrical signal to a touch controller (e.g., a touch IC) or the processor150.

The processor150may be implemented as at least one of a central processing unit, an application processor, or a communication processor and may control the above-mentioned component (e.g., the communication circuitry110, the input/output device120, the memory130, the display140, or the like) of the electronic device100. For example, the processor150may be electrically or operatively connected with at least one component and may deliver at least one instruction associated with a function operation to the component, or may perform various arithmetic operations, a variety of data processing, or the like. In an embodiment, the processor150may map a plurality of data generated or input based on execution of a specified application (e.g., a note application) (e.g., recording data according to recording of a voice input and content data according to a content input) based on a specified criterion (e.g., a time when the plurality of data are generated or input). In an embodiment, the processor150may respond to a user input applied to a screen where the application is executed or set, with reference to the mapped data. For example, in response to a user input for requesting or selecting specific data, concurrently with processing the request or selection, the processor150may visually specify or audibly provide data mapped to the specific data. A description will be given of various function operations of the processor150, associated with this, with reference to the accompanying drawings.

FIGS. 3A to 3Care drawings illustrating types of responses to various user inputs of an electronic device according to an embodiment.FIG. 3Dis a drawing illustrating a form of displaying a graphic effect corresponding to recording playback of an electronic device according to an embodiment.

InFIG. 3A, 3B, 3C, or3D, in a state where a processor (e.g., a processor150ofFIG. 2) of an electronic device100performs mapping of recording data (the recording data may include recorded voice inputs) and content data of a specific event generated or input through execution of a specified application (e.g., a note application). The electronic device100may play the recording data and may output a screen135where an application is executed, which displays the content data, under user control for the application. The screen135where the application is executed, to which other drawings below refer, may be output in the same shape as that described above, and a duplicated description thereof may be omitted.

Referring toFIG. 3A, the processor150of the electronic device100may output the screen135where the application is executed. The screen135can include a first interface10for supporting to output (or play) recording data recorded during a certain time range and a second interface20for displaying content (e.g., text or marking (e.g., an ungrammatical asterisk, underscore, tilde, check, or the like)) input from a user during the certain time range. In an embodiment, the processor150may receive a first user input (e.g., the finger touch) for selecting specific content (“the Creation”) among at least one content displayed on the second interface20. In this case, the processor150may identify at least one stroke to which the first user input is applied and may assign a specified graphic effect (e.g., specified color display, ON/OFF display, or the like) to content of a word, phrase, or morpheme including the at least one identified stroke (e.g., text (the Creation) configured with a plurality of strokes). Alternatively, the processor150may assign the specified graphic effect to content to which the first user input is applied (e.g., marking configured with a single stroke).

In an embodiment, in the operation of assigning the specified graphic effect to the content according to the first user input, the processor150may identify recording data mapped to the content to which the graphic effect is assigned, with reference to a database or index on a memory (e.g., a memory130ofFIG. 2). For example, the processor150may identify recording data recorded at the same time as a time when a stroke is initially input to the content to which the graphic effect is assigned. In an embodiment, the processor150may specify a point where the identified recording data is played on one region of the first interface10.

For example, where the user has made a user input (by their finger) selecting specific content (“the Creation”) among the written script, the processor150may automatically or in response to receiving the user input, move the position of the play bar or marker11to a position within the first interface10associated with a portion of the voice input that was recorded during the time that “the Creation” was written by the user.

In this regard, the first interface10may include a status bar indicating a state where recording data recorded during the certain time is played. The processor150may locate a movement bar11on a point (e.g., 00:02:47) where the identified recording data is played on the status bar. According to an embodiment, the first interface10may include at least one object for controlling to play recording data recorded during the certain time or stop the playback of the recording data. When a user input (e.g., touch) is applied to a play object in the state where the movement bar11is located on the point (e.g., 00:02:47) where the identified recording data is played, the processor150may control an input/output device (e.g., an input/output device120ofFIG. 2) (e.g., a speaker) to output a voice starting from the identified recording data.

In certain embodiments, the opposite can occur. The user can move the movement bar11in first interface10to a position, such as 00:02:47, associated with a portion of the voice input. The processor150will automatically, or in response thereto, highlight the portion of the content, e.g., “the Creation”, which was input at the time associated with 00:02:47.

Referring toFIG. 3B, the processor150may receive a second user input (e.g., drag) for controlling a location of the movement bar11included in the first interface10. In this case, the processor150may identify recording data corresponding to a point (e.g., 00:02:47) where the movement bar11is located, depending on the second user input, with reference to a database or index of the memory130. The processor150may identify content data mapped to the identified recording data (or input at the same time as a time when the identified recording data is recorded) and may assign the specified graphic effect to content21corresponding to the identified content data among at least one content displayed on the second interface20. When a user input (e.g., touch) is applied to a play object included in the first interface10, the processor150may output a voice starting from recording data corresponding to a play point (e.g., 00:02:47) where the movement bar11is located.

Referring toFIG. 3C, a thumbnail object13associated with content data mapped to recording data of a specific play point may be displayed on a region around the status bar included in the first interface10. In an embodiment, when a third user input (e.g., touch) is applied to any one of the at least one thumbnail object13, the processor150may locate the movement bar11on a play point (e.g., 00:57:13) corresponding to a location of the thumbnail object13to which the third user input is applied on the status bar. Furthermore, the processor150may identify content data corresponding to the thumbnail object13to which the third user input is applied (or mapped to recording data of the play point (e.g., 00:57:13) corresponding to a location of the selected thumbnail object13) and may assign the specified graphic effect to content22corresponding to the identified content data among at least one content displayed on the second interface20. When a user input (e.g., touch) is applied to a play button (the triangle pointed to the right) included in the first interface10, the processor150may output a voice starting from recording data corresponding to the play point (e.g., 00:57:13) where the movement bar11is located.

Referring toFIG. 3D, in a state where the movement bar11included in the first interface10is moved to a play point (e.g., 00:02:47) of recording data mapped to content data associated with the first user input according to the above-mentioned first user input (e.g., a touch for selecting specific content among at least one content displayed on the second interface20), the processor150may receive a user input (e.g., touch) applied to a play button15on the first interface10. In this case, the processor150may output a voice starting from recording data corresponding to the play point (e.g., 00:02:47) where the movement bar11is located. In this operation, the processor150may identify content data mapped to recording data corresponding to a point where the movement bar11is moved according to the progress of the voice output, with reference to a database or index on the memory130. The content data may be data that was entered at the time the voice recording of 00:02:47 was recorded. For example, when the movement bar11is moved from a first play point (e.g., 00:02:47) to a second play point (e.g., 00:02:49) depending on the progress of the voice output, the processor150may identify content data mapped to recording data corresponding to the second play point (e.g., 00:02:49). The processor150may assign a specified graphic effect to first content23acorresponding the identified content data among at least one content displayed on the second interface20. After assigning the graphic effect to the first content23a, similarly, the processor150may sequentially assign a specified graphic effect to second content23bcorresponding to a third play point (e.g., 00:02:51) and third content23ccorresponding to a fourth play point (e.g., 00:02:53) as the movement bar11is moved according to the progress of the voice output. According to certain embodiments, the configuration for sequentially assigning the graphic effect to the content23a,23b, and23cdepending on the progress of the voice output is applicable in the same or similar manner when outputting the voice starting from the recording data of the play point where the movement bar11is located inFIG. 3B or 3C.

FIGS. 4 and 5are drawings illustrating types of responses to a fourth user input of an electronic device according to certain embodiments.FIG. 6is a drawing illustrating a type of response to a fifth user input of an electronic device according to an embodiment.

InFIG. 4, the user touches a portion of the content input, e.g., “the Creation” in the second interface. Responsive thereto, the movement bar11moves to a position associated with recorded voice data that was recorded at the time “the Creation” was entered into the second user interface. Additionally, a third interface50appears, converting to text, voice data that was recorded 20 seconds before of “the Creation.” The user can then touch text in the third interface50. Responsive to touching the text in the third interface, such as “In order”, the processor causes the electronic device to play the recorded voice data that resulted in conversion to “In order.”

Referring toFIG. 4, a processor (e.g., a processor150ofFIG. 2) of an electronic device100may receive a fourth user input (e.g., double touch) for selecting specific content (e.g., the Creation) among at least one content displayed on a second interface20. In this case, similar to a response to the above-mentioned first user input (e.g., a touch for selecting specific content among the at least one content displayed on the second interface20), the processor150may locate a movement bar11included in a first interface10on a play point (e.g., 00:02:47) of recording data mapped to content data associated with a fourth user input. Furthermore, the processor150may identify recording data recorded before a specified time (e.g., 20 seconds before) from the recording data mapped to the content data associated with the fourth user input (e.g., recording data corresponding to the play point (00:02:47)) (e.g., recording data corresponding to the play point (00:02:27), with reference to a database or index on a memory (e.g., a memory130ofFIG. 2). The processor150may convert the identified recording data into a text51and may output a third interface50including the converted text51. In an embodiment, the processor150may output a voice starting from the recording data before being converted into the text51in response to a user input (e.g., touch) applied to one region of the third interface50.

InFIG. 5, the user can input content data that includes non-verbal data in the second interface while recording voice data in the first interface10. The user can then select a specific portion of the non-verbal data (the wavelike line). An interface70appears converting to text voice data that was recorded before a specified time of the wavelike line.

Referring toFIG. 5, the processor150may output a screen133where an application (e.g., a note application) is executed, including a first interface10for operating a recording function loaded into the electronic device100and a second interface (not shown) (e.g., a second interface20ofFIG. 1) for entering any information, such as content information. In an embodiment, the processor150may receive a user input for importing a specified file (e.g., a PDF file, an e-mail file, a message file, or the like) using a third menu tab137(e.g., a more menu tab) provided on one region of the screen133. In this case, the processor150may output fourth interface60according to the execution of the file on an upper layer of the second interface (not shown) included in the screen133. According to an embodiment, the interface60may include at least one content (e.g., a text, an image, or the like) according to information included in the file.

In an embodiment, concurrently with recording a voice input received via a microphone in response to user control for the first interface10, the processor150may receive a content input of a user and may display the received content input on the fourth interface60. According to an embodiment, in the embodiment described with reference toFIG. 5, the content input of the user may be understood as a marking input including a non-verbal asterisk, underscore, tilde, check, or the like. The processor150may map the recorded recording data to the input content data on the basis of the same time and may store the mapped data in the form of a database or index.

In an embodiment, the processor150may receive a fourth user input (e.g., double touch) for selecting specific marking content (e.g., tilde marking) among at least one marking content displayed on the fourth interface60. In response to the reception of the fourth user input, the processor150may identify recording data recorded before a specified time (e.g., 20 seconds before) (e.g., recording data recorded at a time (00:10:47) from recording data (e.g., recording data recorded at a time (00:11:07)) mapped to marking content data associated with the fourth user input with reference to the database or index. The processor150may convert the identified recording data into a text71and may output a fifth interface70including the converted text71. In an embodiment, the processor150may output a voice starting from recording data before being converted into the text71in response to a user input (e.g., a touch) applied to a play button15or a user input (e.g., touch) applied to one region of the fifth interface70.

Referring toFIG. 6, the processor150may receive a fifth user input (e.g., drag) for controlling a location of the movement bar11included in the first interface10. In an embodiment, the fifth user input may be understood as an input for returning the movement bar11from a current location to a specific play point (e.g., 00:57:13) at a previous time. The processor150may identify content data mapped to recording data corresponding to the specific play point (e.g., 00:57:13) (or input at the same time as a time when the recording data is recorded). The processor150may assign a specified graphic effect to content22corresponding to the identified content data among at least one content displayed on a second interface20.

In an embodiment, the processor150may receive a user input for controlling new recording from the specific play point (e.g., 00:57:13) (e.g., a long press of greater than or equal to a specified time on a play button included in the first interface10). For example, the processor150may receive a user input for controlling recording of an overwrite mode for a voice input received by the electronic device100, starting from the specific play point (e.g., 00:57:13). Furthermore, concurrently with newly recording the voice input, the processor150may receive a new content input of a user and may display the received content input on the second interface20. In this case, the processor150may identify at least one content data input at a third time corresponding to between a first time corresponding to the specific play point (e.g., 00:57:13) and a second time when the new content input is received, with reference to a database or index of the memory130. The processor150may assign a specified graphic effect (e.g., a dim effect or the like) to content25corresponding to at least one content data input at the third time among at least one content displayed on the second interface20. In other words, except for content24displayed before the first time and content26displayed after the second time, the processor150may assign the specified graphic effect to content25displayed at the third time in a limited manner. Furthermore, the processor150may release mapping between content data input at the third time and recording data recorded at the third time on the database or index.

FIGS. 7A and 7Bare drawings illustrating types of responses to various user inputs of an electronic device according to an embodiment.FIG. 7Cis a drawing illustrating a form of registering specific information associated with processing a seventh user input of an electronic device according to an embodiment.

Referring toFIG. 7A, a screen where the above-mentioned application (e.g., the note application) is executed may include a screen138where the application is set. In an embodiment, a user of an electronic device100may search for some of at least one recording data previously recorded (or content data mapped to the some of the at least one recording data) on the screen138. In this regard, the screen138may support to register a keyword for searching for the some of the at least one recording data. According to an embodiment, the keyword may be understood as content corresponding to content data (for example, an annotated bookmark, “Important”) mapped to the some of the at least one recording data to be searched (e.g., a text in the form of a work, phrase, or morpheme or non-verbal marking (an asterisk, an underscore, a tilde, a check, or the like)). The user may apply an input (e.g., touch) to a fourth menu tab90provided to one region of the screen138to register at least one keyword. A processor150may display at least one keyword80registered from the user on one region of the screen138in the form of a list. The user may select a keyword (e.g., important) corresponding to recording data (or content data mapped to the recording data) to be searched among the at least one content80. In this case, the processor150may identify or collect at least one recording data mapped to content data including the keyword (e.g., important) (or recorded at a time when content data including the keyword is input) (which can referred to as a “voice/content pair”), with reference to a database or index on a memory (e.g., a memory130ofFIG. 2). The processor150may convert the at least one identified or collected recording data into text data81and may display the at least one converted text data81in the form of a list through conversion into another screen138awhere the application is set.

Referring toFIG. 7B, when a user input (e.g., touch) is applied to any one of the at least one text data81displayed in the form of the list, the processor150may identify content data mapped to recording data before being converted into the text data, with reference to the database or text. The processor150may convert the other screen138ainto a screen135where the application is executed and may focus content25corresponding to the identified content data among at least one content displayed on the screen135. Furthermore, the processor150may locate a movement bar11in a first interface10included in the screen135on a play point (e.g., 00:57:13) of recording data mapped to the identified content data. In an embodiment, when a user input (e.g., touch) is applied to a play object in the first interface10, the processor150may output a voice starting from recording data corresponding to the play point (e.g., 00:57:13).

Referring toFIG. 7C, the processor150may provide a fifth menu tab95for supporting to register ungrammatical marking (e.g., an asterisk, an underscore, a tilde, a check, or the like) as the keyword on one region of the screen138where the application is set. In an embodiment, when a user input (e.g., touch) is applied to the fifth menu tab95, the processor150may convert the screen138into the other screen138a. The other screen138amay include, for example, a sixth interface97capable of receiving a content input corresponding to the marking based on a stylus pen (e.g., a stylus pen40ofFIG. 1). In an embodiment, similarly, a keyword according to a content input applied to the sixth interface97may be displayed in the form of a list on one region of the screen138where the application is set.

An electronic device according to certain embodiments may include at least one microphone, a touch screen display, at least one processor configured to be electrically connected with the at least one microphone and the touch screen display, and a memory configured to be electrically connected with the at least one processor.

According to certain embodiments, the memory may store instructions, when executed, causing the at least one processor to output a screen where a specified application is executed, the screen including a first interface for supporting to record a voice input via the at least one microphone and a second interface for supporting a content input of a user, on the touch screen display, map and store voice data recorded at a first time based on the first interface and content data input at the first time based on the second interface in the memory, receive a user input for searching for at least one voice data mapped to specified content data based on a keyword for specified content, collect at least one voice data mapped to the specified content data in response to the user input, convert the at least one collected voice data into text data, and output the converted text data on the touch screen display.

According to certain embodiments, the memory may store further instructions, when executed, causing the at least one processor to display at least one of a status bar indicating a state where recorded voice data is played or a movement bar indicating a point where the recorded voice data is played on the status bar on one region of the first interface.

According to certain embodiments, the memory may further store instructions, when executed, causing the at least one processor to display at least one content according to a content input of the user on one region of the second interface and assign a specified graphic effect to first content in response to a specified first user input for selecting the first content among the at least one content.

According to certain embodiments, the memory may further store instructions, when executed, causing the at least one processor to locate the movement bar on a point where first voice data mapped to first content data associated with the first content is played on the status bar in response to the specified first user input.

According to certain embodiments, the memory may further store instructions, when executed, causing the at least one processor to output a voice starting from a point where the first voice data is played, identify second content data mapped to at least one second voice data recorded at a time after the first voice data, and assign the specified graphic effect to at least one second content associated with the second content data in an order corresponding to the progress of the voice output.

According to certain embodiments, the memory may further store instructions, when executed, causing the at least one processor to display at least one content according to a content input of the user on one region of the second interface, identify third content data mapped to third voice data corresponding to a location of the movement bar in response to a specified second user input for controlling the location of the movement bar, and assign a specified graphic effect to third content associated with the third content data among the at least one content.

According to certain embodiments, the memory may further store instructions, when executed, causing the at least one processor to display at least one thumbnail indicating content data mapped to the recorded voice data on a region around the status bar.

According to certain embodiments, the memory may further store instructions, when executed, causing the at least one processor to display at least one content according to a content input of the user on one region of the second interface and assign a specified graphic effect to fourth content associated with a first thumbnail among the at least one content in response to a specified third user input for selecting the first thumbnail among the at least one thumbnail.

According to certain embodiments, the memory may further store instructions, when executed, causing the at least one processor to display at least one content according to a content input of the user on one region of the second interface and identify fourth voice data recorded before a specified time from recording data mapped to fourth content data associated with fifth content in response to a specified fourth user input for selecting the fifth content among the at least one content.

According to certain embodiments, the memory may further store instructions, when executed, causing the at least one processor to convert the fourth voice data into text data and output a third interface including the converted text data on one region of the screen where the application is executed.

According to certain embodiments, the memory may further store instructions, when executed, causing the at least one processor to display a first object for controlling to play recorded voice data on one region of the first interface and output a voice starting from a point where the fourth voice data is played in response to a specified fifth user input applied to the first object.

According to certain embodiments, the memory may further store instructions, when executed, causing the at least one processor to receive a content input of the user, the content input including at least one of text or ungrammatical marking input using at least one of a software input panel (SIP) keyboard output on the touch screen display and a stylus pen mounted on one region of the electronic device and display at least one content according to the content input on one region of the second interface.

According to certain embodiments, the memory may store instructions, when executed, causing the at least one processor to return the movement bar to a point where the recording voice data is played at a previous time and map and store fifth voice data newly recorded and fifth content data input at the same time as the fifth voice data in response to a specified sixth user input for controlling new recording from the previous time.

According to certain embodiments, the memory may further store instructions, when executed, causing the at least one processor to display at least one content according to a content input of the user on one region of the second interface, identify at least one sixth content data input at a time between the previous time and a time when the fifth content data is input, and assign a specified graphic effect to sixth content corresponding to the sixth content data among the at least one content.

FIG. 8is a flowchart illustrating a method for providing information mapped between a plurality of inputs in an electronic device according to an embodiment. Operations described with reference toFIG. 8may be implemented under control of a processor (e.g., a processor150ofFIG. 2) based on at least one instruction capable of being stored in a memory (e.g., a memory130ofFIG. 2).

In operation801, the processor150of an electronic device (e.g., an electronic device100ofFIG. 2) may execute a specified application (e.g., a note application) in response to user control and may output a first screen where the application is executed. In an embodiment, the first screen may include at least one of a first interface for operating a recording function loaded into the electronic device100or a second interface for supporting a content input of a user.

In operation803, the processor150may map and store voice data recorded at a first time through the first screen and content data input at the first time. In this regard, concurrently with recording a voice input received by the electronic device100in response to user control for the first interface, the processor150may receive a content input of the user and may display the received content input on the second interface. In this operation, the processor150may map the recorded voice data to the input content data on the basis of the same time and may store the mapped data in the form of a database or index.

In operation805, the processor150may receive a user input for searching for specific voice data among the at least one recorded voice data. In this regard, the processor150may output a second screen where the application is executed (e.g., a screen where the application is set) (or switch from the first screen to the second screen) in response to user control and may display at least one keyword registered from the user on one region of the second screen. In an embodiment, the keyword may be understood as content corresponding to content data mapped to voice data to be searched (e.g., a text in the form of a word, phrase, or morpheme or ungrammatical marking (e.g., an asterisk, an underscore, a tilde, a check, or the like)). The user may provide a user input for selecting a keyword corresponding to voice data to be searched among the at least one displayed keyword.

In operation807, the processor150may collect voice data corresponding to the user input. For example, the processor150may identify content data including a keyword selected from the user input with reference to the database or index and may collect at least one voice data mapped to the identified content data.

In operation809, the processor150may output the at least one collected voice data in the form of text. In this regard, the processor150may convert the at least one collected voice data into text data. In an embodiment, the processor150may convert the second screen where the application is executed into a third screen where the application is executed (e.g., another screen where the application is set) and may output the converted text data on one region of the third screen in the form of a list.

According to an embodiment, any one of the at least one text data displayed in the form of the list is selected by the user, the processor150may identify content data mapped to voice data before being converted into the text data, with reference to the database or index. The processor150may convert the third screen into the first screen where the application is executed and may focus content corresponding to the identified content data among at least one content displayed on the first screen.

A method for providing mapped information between a plurality of inputs in an electronic device according to certain embodiments may include outputting a first screen where a specified application is executed, the first screen including a first interface for supporting to record a voice input to the electronic device and a second interface for supporting a content input of a user, mapping and storing voice data recorded at a first time based on the first interface and content data input at the first time based on the second interface, receiving a user input for searching for at least one voice data mapped to specified content data based on a keyword for specified content, collecting at least one voice data mapped to the specified content data, converting the at least one collected voice data into text data, and outputting the converted text data.

According to certain embodiments, the method may further include outputting a second screen where the specified application is executed, the second screen supporting to register the keyword, and providing a menu associated with registering the keyword on one region of the second screen.

According to certain embodiments, the providing of the menu associated with registering the keyword may include displaying at least one keyword registered according to user control through the menu on one region of the second screen.

According to certain embodiments, the receiving of the user input may include receiving a user input for selecting any one of the at least one keyword displayed on the one region of the second screen.

According to certain embodiments, the outputting of the converted text data may include receiving a user input for selecting any one of at least one output text data and converting the second screen into the first screen in response to the user input for selecting the any one of the at least one output text data.

According to certain embodiments, the converting into the first screen may include displaying at least one content corresponding to content data input from the user on one region of the second interface included in the first screen and focusing content corresponding to the selected text data among the at least one content.

FIG. 9is a block diagram illustrating an electronic device901in a network environment900according to certain embodiments.

Referring toFIG. 9, the electronic device901in the network environment900may communicate with an electronic device902via a first network998(e.g., a short-range wireless communication network), or an electronic device904or a server908via a second network999(e.g., a long-range wireless communication network). According to an embodiment, the electronic device901may communicate with the electronic device904via the server908. According to an embodiment, the electronic device901may include a processor920, memory930, an input device950, a sound output device955, a display device960, an audio module970, a sensor module976, an interface977, a haptic module979, a camera module980, a power management module988, a battery989, a communication module990, a subscriber identification module (SIM)996, or an antenna module997. In some embodiments, at least one (e.g., the display device960or the camera module980) of the components may be omitted from the electronic device901, or one or more other components may be added in the electronic device901. In some embodiments, some of the components may be implemented as single integrated circuitry. For example, the sensor module976(e.g., a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented as embedded in the display device960(e.g., a display).

The processor920may execute, for example, software (e.g., a program940) to control at least one other component (e.g., a hardware or software component) of the electronic device901coupled with the processor920, and may perform various data processing or computation. According to one embodiment, as at least part of the data processing or computation, the processor920may load a command or data received from another component (e.g., the sensor module976or the communication module990) in volatile memory932, process the command or the data stored in the volatile memory932, and store resulting data in non-volatile memory934. According to an embodiment, the processor920may include a main processor921(e.g., a central processing unit (CPU) or an application processor (AP)), and an auxiliary processor923(e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor921. Additionally or alternatively, the auxiliary processor923may be adapted to consume less power than the main processor921, or to be specific to a specified function. The auxiliary processor923may be implemented as separate from, or as part of the main processor921.

The auxiliary processor923may control at least some of functions or states related to at least one component (e.g., the display device960, the sensor module976, or the communication module990) among the components of the electronic device901, instead of the main processor921while the main processor921is in an inactive (e.g., sleep) state, or together with the main processor921while the main processor921is in an active state (e.g., executing an application). According to an embodiment, the auxiliary processor923(e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module980or the communication module990) functionally related to the auxiliary processor923.

The memory930may store various data used by at least one component (e.g., the processor920or the sensor module976) of the electronic device901. The various data may include, for example, software (e.g., the program940) and input data or output data for a command related thereto. The memory930may include the volatile memory932or the non-volatile memory934.

The program940may be stored in the memory930as software, and may include, for example, an operating system (OS)942, middleware944, or an application946.

The input device950may receive a command or data to be used by other component (e.g., the processor920) of the electronic device901, from the outside (e.g., a user) of the electronic device901. The input device950may include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus pen).

The sound output device955may output sound signals to the outside of the electronic device901. The sound output device955may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as playing multimedia or playing record, and the receiver may be used for an incoming calls. According to an embodiment, the receiver may be implemented as separate from, or as part of the speaker.

The display device960may visually provide information to the outside (e.g., a user) of the electronic device901. The display device960may include, for example, a display, a hologram device, or a projector and control circuitry to control a corresponding one of the display, hologram device, and projector. According to an embodiment, the display device960may include touch circuitry adapted to detect a touch, or sensor circuitry (e.g., a pressure sensor) adapted to measure the intensity of force incurred by the touch.

The audio module970may convert a sound into an electrical signal and vice versa. According to an embodiment, the audio module970may obtain the sound via the input device950, or output the sound via the sound output device955or a headphone of an external electronic device (e.g., an electronic device902) directly (e.g., wiredly) or wirelessly coupled with the electronic device901.

The interface977may support one or more specified protocols to be used for the electronic device901to be coupled with the external electronic device (e.g., the electronic device902) directly (e.g., wiredly) or wirelessly. According to an embodiment, the interface977may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

A connecting terminal978may include a connector via which the electronic device901may be physically connected with the external electronic device (e.g., the electronic device902). According to an embodiment, the connecting terminal978may include, for example, a HDMI connector, a USB connector, a SD card connector, or an audio connector (e.g., a headphone connector).

The camera module980may capture a still image or moving images. According to an embodiment, the camera module980may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module988may manage power supplied to the electronic device901. According to one embodiment, the power management module988may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

The battery989may supply power to at least one component of the electronic device901. According to an embodiment, the battery989may include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.

The communication module990may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device901and the external electronic device (e.g., the electronic device902, the electronic device904, or the server908) and performing communication via the established communication channel. The communication module990may include one or more communication processors that are operable independently from the processor920(e.g., the application processor (AP)) and supports a direct (e.g., wired) communication or a wireless communication. According to an embodiment, the communication module990may include a wireless communication module992(e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module994(e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate with the external electronic device via the first network998(e.g., a short-range communication network, such as Bluetooth™, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or the second network999(e.g., a long-range communication network, such as a cellular network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multi components (e.g., multi chips) separate from each other. The wireless communication module992may identify and authenticate the electronic device901in a communication network, such as the first network998or the second network999, using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module996.

According to an embodiment, commands or data may be transmitted or received between the electronic device901and the external electronic device904via the server908coupled with the second network999. Each of the electronic devices902and904may be a device of a same type as, or a different type, from the electronic device901. According to an embodiment, all or some of operations to be executed at the electronic device901may be executed at one or more of the external electronic devices902,904, or908. For example, if the electronic device901should perform a function or a service automatically, or in response to a request from a user or another device, the electronic device901, instead of, or in addition to, executing the function or the service, may request the one or more external electronic devices to perform at least part of the function or the service. The one or more external electronic devices receiving the request may perform the at least part of the function or the service requested, or an additional function or an additional service related to the request, and transfer an outcome of the performing to the electronic device901. The electronic device901may provide the outcome, with or without further processing of the outcome, as at least part of a reply to the request. To that end, a cloud computing, distributed computing, or client-server computing technology may be used, for example.

According to certain embodiments, the electronic device may provide information associated with the mapped data in a complex manner by mapping recording data and content data (e.g., text data or the like), which are associated with the same event, depending on a specified criterion.

In addition, various effects ascertained directly or indirectly through the disclosure may be provided.