Patent ID: 12236057

In connection with description of the drawings, identical or similar components may be given identical or similar reference numerals.

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, various embodiments of the present disclosure are disclosed with reference to the accompanying drawings. However, the present disclosure is not intended to be limited by the various embodiments of the present disclosure to a specific embodiment and it is intended that the present disclosure covers all modifications, equivalents, and/or alternatives of the present disclosure provided they come within the scope of the appended claims and their equivalents.

FIG.1is a block diagram of an electronic device101inside a network environment100according to an embodiment.

Referring toFIG.1, the electronic device101in the network environment100may communicate with an electronic device102via a first network198(e.g., a short-range wireless communication network), or at least one of an electronic device104or a server108via a second network199(e.g., a long-range wireless communication network). According to an embodiment, the electronic device101may communicate with the electronic device104via the server108. According to an embodiment, the electronic device101may include a processor120, memory130, an input module150, a sound output module155, a display module160, an audio module170, a sensor module176, an interface177, a connecting terminal178, a haptic module179, a camera module180, a power management module188, a battery189, a communication module190, a subscriber identification module (SIM)196, or an antenna module197. In some embodiments, at least one of the components (e.g., the connecting terminal178) may be omitted from the electronic device101, or one or more other components may be added in the electronic device101. In some embodiments, some of the components (e.g., the sensor module176, the camera module180, or the antenna module197) may be implemented as a single component (e.g., the display module160).

The processor120may execute, for example, software (e.g., a program140) to control at least one other component (e.g., a hardware or software component) of the electronic device101coupled with the processor120, and may perform various data processing or computation. According to one embodiment, as at least part of the data processing or computation, the processor120may store a command or data received from another component (e.g., the sensor module176or the communication module190) in volatile memory132, process the command or the data stored in the volatile memory132, and store resulting data in non-volatile memory134. According to an embodiment, the processor120may include a main processor121(e.g., a central processing unit (CPU) or an application processor (AP)), or an auxiliary processor123(e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor121. For example, when the electronic device101includes the main processor121and the auxiliary processor123, the auxiliary processor123may be adapted to consume less power than the main processor121, or to be specific to a specified function. The auxiliary processor123may be implemented as separate from, or as part of the main processor121.

The auxiliary processor123may control at least some of functions or states related to at least one component (e.g., the display module160, the sensor module176, or the communication module190) among the components of the electronic device101, instead of the main processor121while the main processor121is in an inactive (e.g., sleep) state, or together with the main processor121while the main processor121is in an active state (e.g., executing an application). According to an embodiment, the auxiliary processor123(e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module180or the communication module190) functionally related to the auxiliary processor123. According to an embodiment, the auxiliary processor123(e.g., the neural processing unit) may include a hardware structure specified for artificial intelligence model processing. An artificial intelligence model may be generated by machine learning. Such learning may be performed, e.g., by the electronic device101where the artificial intelligence is performed or via a separate server (e.g., the server108). Learning algorithms may include, but are not limited to, e.g., supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-network or a combination of two or more thereof but is not limited thereto. The artificial intelligence model may, additionally or alternatively, include a software structure other than the hardware structure.

The memory130may store various data used by at least one component (e.g., the processor120or the sensor module176) of the electronic device101. The various data may include, for example, software (e.g., the program140) and input data or output data for a command related thereto. The memory130may include the volatile memory132or the non-volatile memory134.

The program140may be stored in the memory130as software, and may include, for example, an operating system (OS)142, middleware144, or an application146.

The input module150may receive a command or data to be used by another component (e.g., the processor120) of the electronic device101, from the outside (e.g., a user) of the electronic device101. The input module150may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The sound output module155may output sound signals to the outside of the electronic device101. The sound output module155may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as playing multimedia or playing record. The receiver may be used for receiving incoming calls. According to an embodiment, the receiver may be implemented as separate from, or as part of the speaker.

The display module160may visually provide information to the outside (e.g., a user) of the electronic device101. The display module160may include, for example, a display, a hologram device, or a projector and control circuitry to control a corresponding one of the display, hologram device, and projector. According to an embodiment, the display module160may include a touch sensor adapted to detect a touch, or a pressure sensor adapted to measure the intensity of force incurred by the touch.

The audio module170may convert a sound into an electrical signal and vice versa. According to an embodiment, the audio module170may obtain the sound via the input module150, or output the sound via the sound output module155or a headphone of an external electronic device (e.g., an electronic device102) directly (e.g., wiredly) or wirelessly coupled with the electronic device101.

The sensor module176may detect an operational state (e.g., power or temperature) of the electronic device101or an environmental state (e.g., a state of a user) external to the electronic device101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module176may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface177may support one or more specified protocols to be used for the electronic device101to be coupled with the external electronic device (e.g., the electronic device102) directly (e.g., wiredly) or wirelessly. According to an embodiment, the interface177may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

A connecting terminal178may include a connector via which the electronic device101may be physically connected with the external electronic device (e.g., the electronic device102). According to an embodiment, the connecting terminal178may include, for example, a HDMI connector, a USB connector, a SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module179may convert an electrical signal into a mechanical stimulus (e.g., a vibration or a movement) or electrical stimulus which may be recognized by a user via his tactile sensation or kinesthetic sensation. According to an embodiment, the haptic module179may include, for example, a motor, a piezoelectric element, or an electric stimulator.

The camera module180may capture a still image or moving images. According to an embodiment, the camera module180may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module188may manage power supplied to the electronic device101. According to one embodiment, the power management module188may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

The battery189may supply power to at least one component of the electronic device101. According to an embodiment, the battery189may include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.

The communication module190may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device101and the external electronic device (e.g., the electronic device102, the electronic device104, or the server108) and performing communication via the established communication channel. The communication module190may include one or more communication processors that are operable independently from the processor120(e.g., the application processor (AP)) and supports a direct (e.g., wired) communication or a wireless communication. According to an embodiment, the communication module190may include a wireless communication module192(e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module194(e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate with the external electronic device via the first network198(e.g., a short-range communication network, such as Bluetooth™, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or the second network199(e.g., a long-range communication network, such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multi components (e.g., multi chips) separate from each other. The wireless communication module192may identify and authenticate the electronic device101in a communication network, such as the first network198or the second network199, using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module196.

The wireless communication module192may support a 5G network, after a 4G network, and next-generation communication technology, e.g., new radio (NR) access technology. The NR access technology may support enhanced mobile broadband (eMBB), massive machine type communications (mMTC), or ultra-reliable and low-latency communications (URLLC). The wireless communication module192may support a high-frequency band (e.g., the mmWave band) to achieve, e.g., a high data transmission rate. The wireless communication module192may support various technologies for securing performance on a high-frequency band, such as, e.g., beamforming, massive multiple-input and multiple-output (massive MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module192may support various requirements specified in the electronic device101, an external electronic device (e.g., the electronic device104), or a network system (e.g., the second network199). According to an embodiment, the wireless communication module192may support a peak data rate (e.g., 20 Gbps or more) for implementing eMBB, loss coverage (e.g., 164 dB or less) for implementing mMTC, or U-plane latency (e.g., 0.5 ms or less for each of downlink (DL) and uplink (UL), or a round trip of 1 ms or less) for implementing URLLC.

The antenna module197may transmit or receive a signal or power to or from the outside (e.g., the external electronic device) of the electronic device101. According to an embodiment, the antenna module197may include an antenna including a radiating element composed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module197may include a plurality of antennas (e.g., array antennas). In such a case, at least one antenna appropriate for a communication scheme used in the communication network, such as the first network198or the second network199, may be selected, for example, by the communication module190(e.g., the wireless communication module192) from the plurality of antennas. The signal or the power may then be transmitted or received between the communication module190and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element may be additionally formed as part of the antenna module197.

According to various embodiments, the antenna module197may form a mmWave antenna module. According to an embodiment, the mmWave antenna module may include a printed circuit board, a RFIC disposed on a first surface (e.g., the bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a designated high-frequency band (e.g., the mmWave band), and a plurality of antennas (e.g., array antennas) disposed on a second surface (e.g., the top or a side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the designated high-frequency band.

At least some of the above-described components may be coupled mutually and communicate signals (e.g., commands or data) therebetween via an inter-peripheral communication scheme (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).

According to an embodiment, commands or data may be transmitted or received between the electronic device101and the external electronic device104via the server108coupled with the second network199. Each of the electronic devices102or104may be a device of a same type as, or a different type, from the electronic device101. According to an embodiment, all or some of operations to be executed at the electronic device101may be executed at one or more of the external electronic devices102,104, or108. For example, if the electronic device101should perform a function or a service automatically, or in response to a request from a user or another device, the electronic device101, instead of, or in addition to, executing the function or the service, may request the one or more external electronic devices to perform at least part of the function or the service. The one or more external electronic devices receiving the request may perform the at least part of the function or the service requested, or an additional function or an additional service related to the request, and transfer an outcome of the performing to the electronic device101. The electronic device101may provide the outcome, with or without further processing of the outcome, as at least part of a reply to the request. To that end, a cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device101may provide ultra low-latency services using, e.g., distributed computing or mobile edge computing. In another embodiment, the external electronic device104may include an internet-of-things (IoT) device. The server108may be an intelligent server using machine learning and/or a neural network. According to an embodiment, the external electronic device104or the server108may be included in the second network199. The electronic device101may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology or IoT-related technology.

The electronic device according to various embodiments may be one of various types of electronic devices. The electronic devices may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. According to an embodiment of the disclosure, the electronic devices are not limited to those described above.

It should be appreciated that various embodiments of the present disclosure and the terms used therein are not intended to limit the technological features set forth herein to particular embodiments and include various changes, equivalents, or replacements for a corresponding embodiment. With regard to the description of the drawings, similar reference numerals may be used to refer to similar or related elements. It is to be understood that a singular form of a noun corresponding to an item may include one or more of the things, unless the relevant context clearly indicates otherwise. As used herein, each of such phrases as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C,” may include any one of, or all possible combinations of the items enumerated together in a corresponding one of the phrases. As used herein, such terms as “1st” and “2nd,” or “first” and “second” may be used to simply distinguish a corresponding component from another, and does not limit the components in other aspect (e.g., importance or order). It is to be understood that if an element (e.g., a first element) is referred to, with or without the term “operatively” or “communicatively”, as “coupled with,” “coupled to,” “connected with,” or “connected to” another element (e.g., a second element), it means that the element may be coupled with the other element directly (e.g., wiredly), wirelessly, or via a third element.

As used in connection with various embodiments of the disclosure, the term “module” may include a unit implemented in hardware, software, or firmware, and may interchangeably be used with other terms, for example, “logic,” “logic block,” “part,” or “circuitry”. A module may be a single integral component, or a minimum unit or part thereof, adapted to perform one or more functions. For example, according to an embodiment, the module may be implemented in a form of an application-specific integrated circuit (ASIC).

Various embodiments as set forth herein may be implemented as software (e.g., the program140) including one or more instructions that are stored in a storage medium (e.g., internal memory136or external memory138) that is readable by a machine (e.g., the electronic device101). For example, a processor (e.g., the processor120) of the machine (e.g., the electronic device101) may invoke at least one of the one or more instructions stored in the storage medium, and execute it, with or without using one or more other components under the control of the processor. This allows the machine to be operated to perform at least one function according to the at least one instruction invoked. The one or more instructions may include a code generated by a complier or a code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Wherein, the term “non-transitory” simply means that the storage medium is a tangible device, and does not include a signal (e.g., an electromagnetic wave), but this term does not differentiate between where data is semi-permanently stored in the storage medium and where the data is temporarily stored in the storage medium.

According to an embodiment, a method according to various embodiments of the disclosure may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded) online via an application store (e.g., PlayStore™), or between two user devices (e.g., smart phones) directly. If distributed online, at least part of the computer program product may be temporarily generated or at least temporarily stored in the machine-readable storage medium, such as memory of the manufacturer's server, a server of the application store, or a relay server.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single entity or multiple entities, and some of the multiple entities may be separately disposed in different components. According to various embodiments, one or more of the above-described components may be omitted, or one or more other components may be added. Alternatively or additionally, a plurality of components (e.g., modules or programs) may be integrated into a single component. In such a case, according to various embodiments, the integrated component may still perform one or more functions of each of the plurality of components in the same or similar manner as they are performed by a corresponding one of the plurality of components before the integration. According to various embodiments, operations performed by the module, the program, or another component may be carried out sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order or omitted, or one or more other operations may be added.

FIG.2is a block diagram of an electronic device101according to an embodiment.

Referring toFIG.2, the electronic device101may include a processor201, a speaker203, a touch display205, a microphone207, and a memory209. Modules included in the electronic device101may be understood as hardware modules (for example, circuits). Components included in the electronic device101may not be limited to the components illustrated inFIG.2(for example, processor201, speaker203, touch display205, microphone207, and memory209). The components of the electronic device101illustrated inFIG.2may be replaced with other components, or additional components may be added to the electronic device101. For example, at least a part of the configuration of the electronic device101inFIG.1may be applied to the electronic device101inFIG.2.

According to an embodiment, the processor201may execute commands (for example, instructions) stored in the memory209so as to control operations of the components of the electronic device101(for example, processor201, speaker203, touch display205, microphone207, and memory209). The processor201may be connected to the speaker203, the touch display205, the microphone207, and the memory209electrically and/or operatively. The processor201may execute software so as to control at least one other component (for example, speaker203, touch display205, microphone207, and memory209) connected to the processor201. The processor201may acquire a command from components included in the electronic device101, may interpret the acquired command, and may process and/or operate various pieces of data according to the interpreted command.

According to an embodiment, the speaker203may output audio corresponding to audio data. The electronic device101may output audio corresponding to audio data through the speaker203. The electronic device101may output audio corresponding to audio data stored in the electronic device101or audio data acquired from an external device through the speaker203.

According to an embodiment, the touch display205may visually display content. For example, the electronic device101may display at least one of an object, an image, or a video through the touch display205. According to an embodiment, the touch display205may include a touch sensor. The electronic device101may acquire a touch input through the touch display205. For example, the electronic device101may acquire a touch input of a user who inputs an object through the touch display205. In addition, according to an embodiment, a pressure sensor configured to measure the intensity of a force generated by a touch input may be included. In an embodiment, the touch display205may include a display, a hologram device, or a projector and a control circuit for controlling the corresponding device.

According to an embodiment, the touch display205may include a display disposed to be able to slide, thereby providing a screen (for example, display screen). For example, the display area of the electronic device101may be an area exposed visually such that images can be outputted, and the electronic device101may adjust the display area according to movements of a sliding plate (not illustrated) or movements of the display. For example, a rollable-type electronic device may be included, which is configured such that at least a part (for example, housing) of the electronic device101operates to be able to at least partially slide, thereby accomplishing selective expansion of the display area. For example, the touch display205may also be referred to as a slide-out display or an expandable display. For example, the touch display205may change the kind of content provided through the display screen according to a movement (for example, expanded state or reduced state) of the display. For example, first content (for example, object901,902inFIG.9) may be displayed in a first display area in the reduced state, and first content (for example, object901,902inFIG.9) and second content (for example, marking903related to audio data inFIG.9) may be separately displayed in a first display area and in a second display area, respectively, in the expanded state. According to an embodiment, the microphone207may acquire audio. For example, the microphone207may acquire audio corresponding to the user's voice. The electronic device101may acquire audio data corresponding to audio acquired through the microphone207. For example, the electronic device101may acquire audio data corresponding to the user's voice acquired through the microphone207. According to an embodiment, the number of microphones207included in the electronic device101may not be limited. For example, the electronic device101may include multiple microphones207.

According to an embodiment, the electronic device101may, among the multiple microphones207, distinguish and acquire audio data according to the position in which corresponding multiple microphones207are included (disposed) in the electronic device101. For example, multiple microphones207may be disposed at the upper and lower ends of the electronic device101, data acquired through a first microphone disposed at the upper end may be distinguished and stored as first audio data (for example, external audio data), and data acquired through a second microphone disposed at the lower end may be distinguished and stored as second audio data (for example, user audio data).

According to an embodiment, the electronic device101may use the touch display205so as to display an image (or icon) distinguished so as to correspond to audio data acquired through the first microphone and the second microphone. For example, the electronic device101may store data including audio data and a handwriting input, and may display at least one object (for example, object901,902inFIG.9) corresponding to the stored data. The electronic device101may mark an object901including multiple pieces of audio data acquired through multiple microphones, among the at least one object901,902, so as to be distinguished from the other object902. For example, the electronic device101may provide different markings (for example, microphone markings) so as to distinguish first audio data (for example, external audio data) over second audio data (for example, user audio data). In addition, the electronic device101may change, based on a user input regarding each distinguished marking (for example, microphone marking), an area including an object related to audio data (for example, first audio data, second audio data) (for example, operation403inFIG.4). For example, when second audio data among first audio data and second audio data is deactivated (for example, not played back) based on a user input, the electronic device101may deactivate (for example, change transparency of) an area including an object mapped in relation to the second audio data. For example, the electronic device101may distinguish and display the visual effect and/or attribute (for example, thickness, color, or saturation of text) of an object mapped in relation to first audio data and an object mapped in relation to second audio data.

According to an embodiment, the memory209may store various pieces of data used by components of the electronic device101(for example, processor201, speaker203, touch display205, and microphone207) temporarily or non-temporarily. For example, the memory209may store data corresponding to a touch input acquired through the touch display205.

FIG.3is a flowchart300of operations in which an electronic device101according to an embodiment displays an object mapped to an audio playback point-in-time through a first screen.

A series of operations described below may be performed simultaneously by the electronic device101or performed in a changed order, and some operations may be omitted or added.

In operation301, the electronic device101may execute a first application. The first application may be a handwriting software application (or note application), and is not limited to the terminology. For another example, the first application may be understood as a software application including a function for playing audio corresponding to audio data and displaying an object associated with the audio. For still another example, the first application may be understood as a software application including a function for displaying, while audio is played, an object associated with a playback point-in-time of the audio. The object may include at least one of a text, a stroke, an image, a text box, or a web card.

According to an embodiment, the electronic device101may acquire a user input for executing the first application. In response to acquisition of the user input, the electronic device101may execute the first application. The user input may include at least one of a button input, a touch input, a gesture input, a drag input, or a voice input. For example, the electronic device101may acquire the user's touch input for executing the first application through the touch display205. In response to the acquired touch input from the user, the electronic device101may execute the first application. The user input is not limited to the above-mentioned examples, and may include various input methods for executing the first application.

In operation303, in response to execution of the first application, the electronic device101may display a first screen corresponding to the first application. For example, the electronic device101may display a first screen including an area for inputting a handwriting, in response to execution of a handwriting application. For example, the UI of the first screen may be referred toFIG.9. According to an embodiment, in response to execution of the first application, the electronic device101may enter an operation mode for the first application. After entering the mode for the first application, the electronic device101may display a preview screen (for example, UI900inFIG.9) including at least one content (for example, object901,902inFIG.9) corresponding to a data file (for example, memo file) of the first application. The at least one content may include a thumbnail-type image that displays at least a part of the content. The at least one content may include a visual object (for example, marking903related to audio data inFIG.9) according to characteristics of the content. For example, content (for example, object901inFIG.9) including both handwriting data and audio data may include a visual object (for example, marking903related to audio data inFIG.9) indicating audio. In response to acquisition of an input for selecting the at least one content, the electronic device101may display a screen (for example, first screen) corresponding to the at least one content. In operation305, the electronic device101may acquire a first input for playing audio data of the first application. The electronic device101may acquire a first input for playing audio data stored through the first application. For example, the electronic device101may acquire a first input for playing audio data related to a class stored through a handwriting application.

According to an embodiment, the electronic device101may acquire a first input for playing content of the first application. Data of the content may include at least one of audio data or handwriting data. The audio data and the handwriting data may be associated based on time information. For example, a playback point-in-time at which audio corresponding to the audio is played and a handwriting corresponding to the handwriting data may be mapped.

In operation307, in response to acquisition of the first input for playing audio data of the first application, the electronic device101may output audio corresponding to the audio data through the speaker203. For example, in response to acquisition of a user input for playing content of the handwriting application, the electronic device101may output audio corresponding to audio data related to a class through the speaker203.

In operation309, the electronic device101may determine whether an object mapped to the playback point-in-time of the audio is displayed in the first screen currently displayed on the display. According to an embodiment, the electronic device101may identify whether an object associated with the audio exists. When the object exists, the electronic device101may determine the position of the identified object. Based on the position of the identified object, the electronic device101may determine whether the object is displayed in the first screen. The first screen may be the entire area or partial area (for example, area in which text is displayed) of the current screen displayed in response to execution of the first application. For example, the electronic device101may identify whether a handwriting mapped to a playback point-in-time (for example, a point-in-time of three minutes that lapses from the starting time of the audio) of the audio exists. When the handwriting exists, the electronic device101may determine the position of the identified handwriting. Based on the determined position, the electronic device101may determine whether the handwriting displayed in the first screen.

According to an embodiment, the electronic device101may display an object mapped to a playback point-in-time of the audio on the first screen or may display the object after changing the attribute (for example, thickness, color, or saturation of text) of the object. For example, the electronic device101may change the color of the text mapped to a playback point-in-time of the audio from a vague color to a dark color and then display the color text. According to an embodiment, when the object is displayed in the first screen, the electronic device101may perform operation311and, when the object is not displayed in the first screen, the electronic device101may perform operation313.

In operation311, when an object mapped to a playback point-in-time of the audio is displayed in the first screen, the electronic device101may omit auto scroll regarding the object. For example, when audio related to a class is being output, and if a handwriting mapped to a playback point-in-time of the audio is displayed in the first screen, the electronic device101may omit auto scroll regarding the handwriting and may maintain the first screen.

In operation313, the electronic device101may perform auto scroll such that the first screen is changed to a second screen including an identified object. The first screen and the second screen may be understood as screens displayed in response to selection of an object corresponding to identical data (for example, identical document file) stored in the first application. The first screen and the second screen may be at least partially overlap or may not overlap. In the process of changing from the first screen to the second screen through the auto scroll, the first screen may be moved while being automatically scrolled upwards or downwards. The auto scroll may be understood as a function for performing scroll automatically such that the screen displayed by the electronic device101is changed. For example, when an object mapped to an audio playback point-in-time is not displayed on the first screen, the electronic device101may change the position of the area displayed on the touch display205to a second screen including the object, through auto scroll. For example, when a screen including an object mapped to an audio playback point-in-time is positioned below the screen currently displayed on the display, the electronic device101may move the screen currently displayed on the display upwards through auto scroll such that the screen including the object is displayed through the touch display205by such conversion. As another example, when a screen including an object mapped to an audio playback point-in-time is positioned above the screen currently displayed on the display, the electronic device101may move the screen currently displayed on the display downwards through auto scroll such that the screen including the object is displayed through the touch display205by such conversion.

FIG.4is a flowchart400of operations in which an electronic device101according to an embodiment determines whether or not to perform auto scroll, based on an area including an object.

A series of operations described below may be performed simultaneously by the electronic device101or performed in a changed order, and some operations may be omitted or added.

Referring toFIG.4, the electronic device101may determine an area including an object mapped to a playback point-in-time of an outputted audio. The electronic device101may determine whether or not to perform auto scroll, based on whether the determined area is displayed on a first screen.

In operation401, the electronic device101may identify whether an object mapped to an audio playback point-in-time exists. According to an embodiment, when the identified object exists, the electronic device101may perform operation403and, when the identified object does not exist, the electronic device101may perform operation307.

In operation403, when an object mapped to the audio playback point-in-time exists, the electronic device101may determine an area including the object mapped to the audio playback point-in-time. The electronic device101may determine at least one of the position of the determined object or the position of the determined area. For example, the electronic device101may configure a rectangular area including the determined area and may determine the center of the rectangular area as the position of the rectangular area. The position of the area is not limited to the center of the area, and may be determined to be any point in the area. The electronic device101may acquire the x-coordinate and y-coordinate of the area, and may determine the position of the area, based on the acquired coordinates. The electronic device101may determine, based on the determined area position, whether the determined area is positioned above or below the first screen. The area may refer to an area surrounding the object. For example, the area may be a quadrangular area surrounding the object. The shape is not limited to the above-mentioned examples.

In operation405, based on the determined area position, the electronic device101may determine whether the area including the object mapped to the audio playback point-in-time is included in the first screen. When the x-coordinate and y-coordinate of the determined area are included in the coordinate range of the first screen, the electronic device101may determine that the determined area is included in the first screen. The coordinate range may be configured with reference to coordinate (0, 0) of the left lower end of the first screen. For example, when the coordinate of the determined area is (3, 5), and when the coordinate range of the first screen is (0, 0) to (50, 70), the electronic device101may determine that the determined area is included in the first screen. The coordinate range is not limited to the above-mentioned examples. For another example, the coordinate range may be configured with reference to coordinate (0, 0) of the left upper end of the first screen. For still another example, when the coordinate of the determined area is (−3, −5), and when the coordinate range of the first screen is (0, 0) to (−50, −70), the electronic device101may determine that the determined area is included in the first screen.

According to an embodiment, when the determined area is included in the first screen, the electronic device101may perform operation407and, when the determined area is not included in the first screen, the electronic device101may perform operation409.

In operation407, when determined area is included in the first screen, the electronic device101may omit performing auto scroll and maintain the first screen.

In operation409, when the determined area is not included in the first screen, the electronic device101may perform auto scroll such that the screen currently displayed on the display is changed from the first screen to a second screen including the determined area. For example, when the determined area is positioned above the first screen, the electronic device101may move the first screen downwards through auto scroll such that the screen currently displayed on the display is changed from the first screen to a second screen including the determined area. As another example, when the determined area is below above the first screen, the electronic device101may move the first screen upwards through auto scroll such that the screen currently displayed on the display is changed from the first screen to a second screen including the determined area. For example, it may be determined whether the determined area is positioned above or below the first screen in view of the y-coordinate of the determined area and the coordinate range of the first screen. The first screen and the second screen may be understood as screens displayed in response to selection of an object corresponding to identical data (for example, identical document file) stored in the first application. According to an embodiment, when a second area including a second object mapped to a second audio playback point-in-time is positioned below a first area including a first object mapped to a first audio playback point-in-time, the electronic device101may move the first screen including the first area upwards through auto scroll such that the same changes to the second screen including the second area. As another example, when a second area including a second object mapped to a second audio playback point-in-time is positioned above a first area including a first object mapped to a first audio playback point-in-time, the electronic device101may move the first screen including the first area downwards through auto scroll such that the same changes to the second screen including the second area. The second playback point-in-time may be understood as a point-in-time after a time lapse from the first playback point-in-time.

FIG.5is a flowchart500of operations in which an electronic device101according to an embodiment minimizes screen movements through auto scroll.

A series of operations described below may be performed simultaneously by the electronic device101or performed in a changed order, and some operations may be omitted or added.

In operation501, the electronic device101may display a first object mapped to a first audio playback point-in-time in a first screen. The operation of displaying the first object in the first screen may be performed after operation311or313inFIG.3is performed. For example, the first object may be displayed in the first screen by performing auto scroll or without performing the auto scroll.

In operation503, the electronic device101may determine whether a second object mapped to a second audio playback point-in-time is displayed in the first screen. According to an embodiment, when the second object is displayed in the first screen, the electronic device101may perform operation505and, when the second object is not displayed in the first screen, the electronic device101may perform operation507.

In operation505, when the second object is displayed in the first screen, the electronic device101may omit performing auto scroll such that the first screen keeps displaying the second object. The electronic device101may omit performing the auto scroll and may maintain the first screen.

In operation507, when the second object is not displayed in the first screen, the electronic device101may determine whether the first and second objects can be displayed in a single screen. For example, when the distance between the position of a first object mapped to a first playback point-in-time of audio and the position of a second object mapped to a second playback point-in-time of the audio is less than the size of the first screen, the first and second objects may be displayed in a single screen.

According to an embodiment, when multiple objects including first and second objects can be displayed in a first screen, the electronic device101may display the first object in the first screen around an object (for example, second object) related to a playback point-in-time (for example, second playback point-in-time) after a time lapse. For example, the electronic device101may display the second object at the center of the first screen and may display the first object at the upper end of the first screen.

According to an embodiment, when the first and second objects can be displayed in a single screen, the electronic device101may perform operation509and, when the first and second objects cannot be displayed in a single screen, the electronic device101may perform operation511.

In operation509, when the first and second objects can be displayed in a single screen, the electronic device101may perform auto scroll such that the first screen is changed to a second screen including the first and second objects. When the first and second objects can be displayed in a single screen, the electronic device101may change the first screen to the second screen including the first and second objects by performing the auto scroll.

In operation511, when the first and second objects cannot be displayed in a single screen, the electronic device101may perform auto scroll such that the first screen is changed to a third screen including the second object, not the first object. Operation511inFIG.5may correspond to operation313inFIG.3and operation409inFIG.4.

FIG.6is a flowchart600of operations in which an electronic device101according to an embodiment generates playback data.

A series of operations described below may be performed simultaneously by the electronic device101or performed in a changed order, and some operations may be omitted or added.

In operation601, the electronic device101may execute a first application. The first application may be understood as a handwriting application, and is not limited to the terminology. For another example, the first application may be understood as an application including a function for playing audio corresponding to audio data and displaying an object associated with the audio. Operation601inFIG.6may correspond to operation301inFIG.3.

In operation603, the electronic device101may display a first screen corresponding to the first application. Operation603inFIG.6may correspond to operation303inFIG.3.

In operation605, the electronic device101may acquire audio data corresponding to audio acquired through the microphone207. While displaying the first screen, the electronic device101may acquire an event that starts an audio input. The event may include an input for selecting an object displayed on the first screen. In response to acquisition of the event, the electronic device101may acquire audio through the microphone207. For example, in response to acquisition of the event, the electronic device101may acquire audio generated during a lecture through the microphone207.

In operation607, the electronic device101may acquire object data (e.g., text data) corresponding to an object mapped to an input point-in-time of the acquired audio. While the electronic device101acquires audio through the microphone207, the electronic device101may acquire a handwriting input through the touch display205. The acquired audio and the acquired handwriting input may be associated in relation to an input point-in-time of the acquired audio. For example, while the electronic device101acquires audio through the microphone207, the electronic device101may acquire an object mapped to an input point-in-time (for example, a point-in-time of three minutes that lapses from the starting time of the audio) of the acquired audio through the touch display205. The electronic device101may acquire object data corresponding to the acquired object.

In operation609, the electronic device101may identify an event that ends acquisition of the audio and object (e.g., text). The ending event may include an input for selecting an object displayed on the first screen. In response to acquisition of an input for selecting an object displayed on the first screen, the electronic device101may sense an event that ends acquisition of the audio and object.

In operation611, the electronic device101may generate playback data including audio data and object data mapped to an input point-in-time of the audio data. For example, while outputting audio, the electronic device101may generate playback data including content including an object mapped to an input point-in-time of the outputted audio. The electronic device101may store the generated playback data.

FIG.7is a flowchart700of operations in which an electronic device101according to an embodiment synthesizes playback data.

A series of operations described below may be performed simultaneously by the electronic device101or performed in a changed order, and some operations may be omitted or added.

In operation701, the electronic device101may generate first playback data. In response to identifying an event that ends acquisition of audio and object inputs (operation609inFIG.6), the electronic device101may generate first playback data including audio data and object data mapped to an input point-in-time of audio data. The electronic device101may store the generated first playback data. Operation701inFIG.7may correspond to operation611inFIG.6.

In operation703, the electronic device101may generate second playback data. The operation of generating the second playback data may be substantially identical to the operation of generating the first playback data. The first playback data may be understood as first audio and handwriting data stored in a first handwriting file of a handwriting application, and the second playback data may be understood as second audio and handwriting data stored in the first handwriting file of the first application.FIG.7illustrates two pieces of playback data generating operations, but is not limited thereto, and multiple pieces of playback data including third playback data may be generated.

In operation705, the electronic device101may acquire a user input for synthesizing playback data. For example, when multiple pieces of playback data are stored in the first handwriting file of the handwriting application, a user input for synthesizing the multiple pieces of playback data may be acquired.

In operation707, in response to acquiring the user input for synthesizing playback data, the electronic device101may synthesize first playback data and second playback data. Synthesis of pieces of playback data may refer to converting multiple pieces of playback data into a single piece of playback data. According to an embodiment, when first playback data, second playback data, and third playback data are stored in the first handwriting file of the first application, the electronic device101may synthesize the first playback data, the second playback data, and the third playback data.

FIGS.8A and8Billustrate a user interface (UI) of a first screen801according to an embodiment.

FIG.8Aillustrates a first screen801displayed in response to execution of a first application.FIG.8Billustrates a first screen801displayed in response to a first input for playing audio data.

Referring toFIG.8AandFIG.8B, the electronic device101may display a first screen801in response to execution of a first application. For example, the electronic device101may display a first screen801including an input area for inputting content (for example, image815, handwriting813, or text811) in response to execution of a handwriting application.

According to an embodiment, the first screen801may include a menu803, an input area for inputting content (for example, image815, handwriting813, or text811), and an object805related to audio playback. The menu803may include an image menu for adding an image to the input area, a voice recording menu for recording a voice, and an audio file menu for adding an audio file. The menu803may include various menus in addition to the above-mentioned menus. The electronic device101may acquire the user's handwriting or image through the input area. The object805related to audio playback may include an object for selecting a playback point-in-time (for example, two seconds) of audio data, or for stopping or pausing playback of the audio data.

FIG.9illustrates a UI900related to a first screen801according to an embodiment.

Referring toFIG.9, in response to execution of the first application, the electronic device101may display a UI900including at least one object901,902corresponding to stored data. The at least one object may include a thumbnail-type image which displays at least a part of the stored data. Based on the characteristics of the stored data, the at least one object901,902may have different types. For example, when the stored data includes audio data, the at least one object901may include a marking903(for example, microphone marking) related to the audio data. In response to a user input for selecting the at least one object901,902, the electronic device101may display a first screen801corresponding to the at least one object901,902. According to an embodiment, the electronic device101may display the at least one object901,902in various formats. For example, the electronic device101may display the at least one object901,902in a grid, list, or simple list format. In the grid format, the object901,902may be displayed in a symmetric format (for example, 2×2, 3×3) or in an asymmetric format based on the size of a thumbnail according to the size of the object901,902. In the list format, the object901,902may be displayed so as to include a large thumbnail, the date and/or time at which the large thumbnail is generated. In the simple list format, the object901,902may be displayed so as to include a small thumbnail, a title, the date and/or time at which the small thumbnail is generated. According to an embodiment, in response to acquiring an input for selecting an object from at least one object displayed in the grid format, the electronic device101may display a first screen801corresponding to the selected object.

FIG.10AtoFIG.10Dillustrate a UI related to text display using auto scroll while outputting audio according to an embodiment.

Referring toFIG.10AtoFIG.10D, while outputting audio, the electronic device101may display a text mapped to a playback point-in-time of the audio through a first screen801by using auto scroll.

FIG.10Aillustrates a first screen801displayed before audio data is played. The first screen801may include an object1005related to audio playback, an input area1011, and a marking of a search function. The object1005related to audio playback may include a marking1001indicating an audio playback point and a marking1003indicating audio playback, stop, and pause. The electronic device101may acquire a text1021through the input area1011, and may display an image1013and a text1021.

According to an embodiment, the electronic device101may acquire handwriting data corresponding to a handwriting acquired through the first screen801. The electronic device101may determine a text corresponding to the acquired handwriting data. The electronic device101may convert the acquired handwriting data into a text corresponding to the handwriting data. The electronic device101may assign a visual effect to the handwriting data or the text and then display the handwriting data or the text.

According to an embodiment, the electronic device101may acquire audio data corresponding to audio acquired through the microphone207. The electronic device101may determine audio data corresponding to the acquired audio.

According to an embodiment, the electronic device101may acquire a user input for a text search and, in response to the user input, may determine the searched text among texts stored in a handwriting file of a first application. The electronic device101may visually display the determined text.

FIG.10Billustrates the first screen801displayed while audio data is played. The object1005related to audio playback may indicate a first playback point-in-time1001-1(for example, a point-in-time after 13 seconds lapses from the starting time of the audio) of the audio. While the audio is outputted, the electronic device101may display a text1021-1mapped to the first playback point-in-time1001-1of the outputted audio through the first screen801.

FIG.10Cillustrates the first screen801displayed while audio data is played. The object1005related to audio playback may indicate a second playback point-in-time1001-2(for example, a point-in-time of 6 minutes 37 seconds that lapses from the starting time of the audio) of the audio. While the audio is outputted, the electronic device101may determine whether a text1021-2mapped to the second playback point-in-time1001-2of the outputted audio is displayed on the first screen801. When the text1021-2is not displayed on the first screen801, the electronic device101may convert the screen through auto scroll such that the text1021-2is displayed on the first screen801. For example, the first screen801illustrated inFIG.10Bdoes not display the second text1021-2mapped to the second playback point-in-time1001-2of the audio, and the electronic device101may thus convert the screen to the first screen801illustrated inFIG.10Cby using auto scroll such that the second text1021-2mapped to the second playback point-in-time1001-2of the audio is displayed on the first screen801.

FIG.10Dillustrates the first screen801displayed after audio data is played. The object1005related to audio playback may indicate a third playback point-in-time1001-3(for example, a point-in-time of 40 minutes 12 seconds that lapses from the starting time of the audio) of the audio. While the audio is outputted, the electronic device101may determine whether a text1021-3mapped to the third playback point-in-time1001-3of the outputted audio is displayed on the first screen801. When the text1021-3is not displayed on the first screen, the electronic device101may convert the screen through auto scroll such that the text1021-3is displayed on the first screen801. For example, the first screen801illustrated inFIG.10Cdoes not display the third text1021-3mapped to the third playback point-in-time1001-3of the audio, and the electronic device101may thus convert the screen to the first screen801illustrated inFIG.10Dby using auto scroll such that the third text1021-3mapped to the third playback point-in-time1001-3of the audio is displayed on the first screen801at the third playback point-in-time1001-3.

According to an embodiment, the electronic device101may determine whether an image displayed in the first screen801is associated with an object (for example, text). When content (for example, image and text) of the first application is generated and stored, association between the content (for example, image and text) may be together determined and stored. When the image and the object are associated, the electronic device101may fix the image in an area in the first screen801and may perform auto scroll with regard to the object, thereby converting the first screen801. When the image and the object are not associated, the electronic device101may perform auto scroll with regard to the image and the object, thereby converting the first screen801.

FIGS.11A and11Billustrate a UI related to synthesis of playback data according to an embodiment.

Referring toFIG.11A, the electronic device101may synthesize multiple pieces of playback data (for example, Voice0011101, Voice0021102, and Voice0031103).

FIG.11Aillustrates a first screen801on which multiple pieces of playback data are displayed. The audio playback point1100may indicate the current playback point-in-time among the entire playback time of audio data.

According to an embodiment, the first point-in-time1101-1may correspond to a point-in-time at which playback of first playback data (for example, Voice0011101) starts, the second point-in-time1101-2may correspond to a point-in-time at which playback of second playback data (for example, Voice0021102) starts, and the third point-in-time1101-3may correspond to a point-in-time at which playback of third playback data (for example, Voice0031103) starts.

According to an embodiment, the electronic device101may acquire a user input for playing playback data. In response to acquisition of the user input, the electronic device101may successively play multiple pieces of playback data. For example, in response to acquisition of the user input, the electronic device101may successively play first playback data (for example, Voice0011101), second playback data (for example, Voice0021102), and third playback data (for example, Voice0031103).

According to an embodiment, in response to acquisition of an input for selecting a point corresponding to playback data, the electronic device101may play the playback data. For example, in response to acquisition of an input for selecting a second point-in-time1102-1corresponding to second playback data (for example, Voice0021102), the electronic device101may play the second playback data.

FIG.11Billustrates a first screen801for displaying a single piece of playback data (for example, Voice0011104) obtained by synthesizing multiple pieces of playback data (for example, Voice0011101, Voice0021102, and Voice0031103).

According to an embodiment, the electronic device101may acquire a user input for synthesizing playback data. For example, when multiple pieces of playback data (for example, Voice0011101, Voice0021102, and Voice0031103) are stored in a first handwriting file of a first application, a user input for synthesizing the multiple pieces of playback data may be acquired. In response to acquisition of a user input for synthesizing the pieces of playback data, the electronic device101may synthesize the multiple pieces of playback data. Synthesis of the pieces of playback data may correspond to conversion of multiple pieces of playback data (for example, Voice0011101, Voice0021102, and Voice0031103) into a single piece of playback data (for example, Voice0011104).

Referring toFIG.11A, when multiple pieces of playback data (for example, Voice0011101, Voice0021102, and Voice0031103) exist in a first handwriting file, the electronic device101may respond to a user input for selecting a piece of playback data (for example, Voice0011101) from the multiple pieces of playback data, may apply a visual effect (for example, dimming) to content mapped to the piece of playback data (for example, Voice0011101), and may display the same accordingly. For example, the electronic device101may apply a visual effect to content corresponding to data selected from the multiple pieces of playback data and may apply no visual effect to content corresponding to unselected data.

Referring toFIG.11B, when a single piece of playback data (for example, Voice0011104) into which multiple pieces of playback data (for example, Voice0011101, Voice0021102, and Voice0031103) are synthesized exists in a first handwriting file, the electronic device101may respond to acquisition of a user input for selecting the single piece of playback data, may apply a visual effect (for example, dimming) to content mapped to the single piece of playback data (for example, Voice0011101), and may display the same accordingly.

According to an embodiment, the electronic device101may align time information (for example, information regarding the time during which playback data has been recorded) including the start, pause, resume, and stop of multiple pieces of playback data, thereby combining the same into a single piece of playback data. Based on alignment of the time information, the electronic device101may combine multiple pieces of playback data into a single piece of playback data corresponding to a time flow.

According to an embodiment, an electronic device101may include: a speaker203, a touch display205, and a processor201operatively connected to the speaker203and the touch display205. The processor201may execute a predetermined software application, display a first screen corresponding to the predetermined software application through the touch display205in response to the execution of the predetermined software application, acquire a first input for playing audio data of the predetermined software application while the first screen is displayed, output audio corresponding to the audio data through the speaker203in response to the acquisition of the first input, determine whether an object mapped to a playback point-in-time of the outputted audio is displayed in the first screen, and perform auto scroll, when the object is not displayed in the first screen, such that the object is displayed through the first screen while the audio is outputted.

According to an embodiment, the processor201of the electronic device101may display the object mapped to the playback point-in-time of the outputted audio through the first screen, while the audio is outputted, in response to the performing of the auto scroll.

According to an embodiment, the processor201of the electronic device101may display the object mapped to the playback point-in-time of the outputted audio through the first screen by using a visual effect.

According to an embodiment, the processor201of the electronic device101may determine an area including the object mapped to the playback point-in-time of the outputted audio, determine whether the determined area is included in the first screen, and perform the auto scroll, when the determined area is not included in the first screen, such that the determined area is included in the first screen while the audio is outputted.

According to an embodiment, the processor201of the electronic device101may display the object included in the determined area and mapped to the playback point-in-time of the outputted audio through the first screen, while the audio is outputted, in response to the performing of the auto scroll.

According to an embodiment, the object may include a first object and a second object, and the playback time-point may include a first playback time-point and a second playback time-point, and the processor201of the electronic device101may display the first object mapped to the first playback point-in-time of the outputted audio through the first screen while the audio is outputted, determine, when the second object mapped to the second playback point-in-time of the outputted audio is not displayed in the first screen, whether the first object and the second object can be displayed in a single screen, and perform the auto scroll, when the first object and the second object can be displayed in a single screen, such that the first screen is converted to a second screen in which the first object and the second object are included.

According to an embodiment, the processor201of the electronic device101may perform the auto scroll, when the first object and the second object cannot be displayed in a single screen, such that the first screen is converted into a third screen in which the second object is included.

According to an embodiment, the processor201of the electronic device101may perform the auto scroll, when a part of the object is not displayed in the first screen, such that the object is displayed through the first screen while the audio is outputted.

According to an embodiment, the processor201of the electronic device101may determine whether the object mapped to the playback point-in-time of the outputted audio exists, determine a position of the object when the object exists, and determine, based on the determined position, whether the object is displayed in the first screen.

According to an embodiment, the object may include a first object and a second object, and the playback time-point may include a first playback time-point and a second playback time-point, and the processor201of the electronic device101may acquire a user input for fixing the first object mapped to the first playback point-in-time of the outputted audio in a first area of the first screen, display the first object in the first area of the first screen in response to the acquisition of the user input, and perform the auto scroll, when the second object mapped to the second playback point-in-time of the outputted audio data is not displayed in a second area of the first screen, such that the second object is displayed through the second area of the first screen while the audio is outputted.

According to an embodiment, an electronic device101may include a microphone207, a touch display205, and a processor201operatively connected to the microphone207and the touch display205. The processor201may execute a predetermined software application, display a first screen corresponding to the predetermined software application through the touch display205in response to the execution of the predetermined software application, acquire audio data corresponding to audio acquired through the microphone207while the first screen is displayed, acquire object data corresponding to an object mapped to an input point-in-time of the acquired audio through the touch display205while the first screen is displayed, identify an event for ending the acquisition of the audio data and the object data, and generate first playback data including the audio data and the object data mapped to the input point-in-time of the audio data in response to identifying the event for ending the acquisition.

According to an embodiment, wherein the object may include a first object and a second object, and the playback data may include first playback data and second playback data, and the processor201of the electronic device101may display the first object indicating the first playback data and the second object indicating the second playback data in the first screen, acquire a first input for synthesizing the first playback data and the second playback data through the touch display205, synthesize the first playback data and the second playback data, in response to the acquisition of the first input, thereby generating third playback data, and display a third object corresponding to the third playback data in the first screen.

According to an embodiment, a method for operating an electronic device101may include: the operations of executing a predetermined software application, displaying a first screen corresponding to the predetermined software application through a touch display205in response to the execution of the predetermined software application, acquiring a first input for playing audio data of the predetermined software application while the first screen is displayed, outputting audio corresponding to the audio data through a speaker in response to the acquisition of the first input, determining whether an object mapped to a playback point-in-time of the outputted audio is displayed in the first screen, and performing auto scroll, when the object is not displayed in the first screen, such that the object is displayed through the first screen while the audio is outputted.

According to an embodiment, the method for operating an electronic device101may further include an operation of displaying the object mapped to the playback point-in-time of the outputted audio through the first screen, while the audio is outputted, in response to the performing of the auto scroll.

According to an embodiment, the method for operating an electronic device101may further include an operation of displaying the object mapped to the playback point-in-time of the outputted audio through the first screen by using a visual effect.

According to an embodiment, the method for operating an electronic device101may further include the operations of determining an area including an object mapped to a playback point-in-time of the outputted audio, determining whether the determined area is included in the first screen, and performing auto scroll, when the determined area is not included in the first screen, such that the determined area is included in the first screen while the audio is outputted.

According to an embodiment, the method for operating an electronic device101may further include an operation of displaying an object included in the determined area and mapped to a playback point-in-time of the outputted audio through the first screen, while the audio is outputted, in response to performing of the auto scroll.

According to an embodiment, the method for operating an electronic device101may further include the operations of displaying a first object mapped to a first playback point-in-time of the outputted audio through the first screen while the audio is outputted, determining, when a second object mapped to a second playback point-in-time of the outputted audio is not displayed in the first screen, whether the first object and the second object can be displayed in a single screen, and performing the auto scroll, when the first object and the second object can be displayed in a single screen, such that the first screen is converted to a second screen in which the first object and the second object are included.

According to an embodiment, the method for operating an electronic device101may further include the operations of determining whether an object mapped to a playback point-in-time of the outputted audio exists, determining a position of the object when the object exists, and determining, based on the determined position, whether the object is displayed in the first screen.

According to an embodiment, the method for operating an electronic device101may further include the operations of executing a predetermined software application, displaying a first screen corresponding to the predetermined software application through a touch display205in response to the execution of the predetermined software application, acquiring audio data corresponding to audio acquired through a microphone while the first screen is displayed, acquiring object data corresponding to an object mapped to an input point-in-time of the acquired audio through the touch display205while the first screen is displayed, identifying an event for ending the acquisition of the audio and the object, and generating playback data including the audio data and the object data mapped to an input point-in-time of the audio data in response to identifying the event for ending the acquisition.