Patent Description:
A remote controller is mainly used as an input interface for a conventional display device. That is, when a user requests a function by pressing various input buttons provided on the remote control, the display device performs an operation corresponding to the request.

With the rapid development of digital technology, as content is produced in high-spec and high-definition, the display device is also developing accordingly. Conventionally, it is inconvenient to control the display device as desired by a user through a button provided in the remote control or through gesture recognition through the remote control.

Accordingly, input technology through voice recognition has been adopted in display devices. However, the conventional voice recognition technology has difficulty in accurately recognizing a user's voice in various situations. In particular, for example, when an unintended sound source is recognized because a nearby interference sound source is introduced upon voice recognition, there is a problem of performing the voice recognition process repeatedly.

In order to solve this problem, there is a method of removing a small level of input source when a level difference between input sound sources is large as one of the methods of removing the interference sound source, but there is still a limit. <CIT> relates to a conversation support apparatus. <CIT> relates to methods and systems for automatic transcription of a conversation.

The invention is defined by independent claims <NUM> and <NUM>. An object of the present disclosure is to perform voice recognition processing by identifying a plurality of speakers in a display device when the speakers speak.

Another object of the present disclosure is to provide a service that matches a user's intention for a display device through voice recognition processing based on information on identification of speakers to increase convenience of use and product satisfaction.

The suffixes "module" and "unit or portion" for components used in the following description are merely provided only for facilitation of preparing this specification, and thus they are not granted a specific meaning or function.

A display device suitable to an embodiment of the present disclosure is, for example, an intelligent display device in which a computer support function is added to a broadcast reception function, and is faithful to a broadcast reception function and has an Internet function added thereto, such as a handwritten input device, a touch screen Alternatively, a more user-friendly interface such as a spatial remote control may be provided. In addition, it is connected to the Internet and a computer with the support of a wired or wireless Internet function, so that functions such as e-mail, web browsing, banking, or games can also be performed. A standardized general-purpose OS may be used for these various functions.

Accordingly, in the display device described in the present disclosure, various user-friendly functions can be performed because various applications can be freely added or deleted, for example, on a general-purpose OS kernel. More specifically, the display device may be, for example, a network TV, HBBTV, smart TV, LED TV, OLED TV, and the like, and may be applied to a smart phone in some cases.

<FIG> is a block diagram illustrating a configuration of a display device according to an embodiment, not encompassed by the wording of the claims but useful for understanding the context of the invention.

Referring to <FIG>, a display device <NUM> may include a broadcast receiver <NUM>, an external device interface <NUM>, a memory <NUM>, a user input interface <NUM>, a controller <NUM>, a wireless communication interface <NUM>, a microphone <NUM>, a display <NUM>, a speaker <NUM>, and a power supply circuit <NUM>.

The broadcast receiver <NUM> may include a tuner <NUM>, a demodulator <NUM>, and a network interface <NUM>.

The tuner <NUM> may select a specific broadcast channel according to a channel selection command. The tuner <NUM> may receive a broadcast signal for the selected specific broadcast channel.

The demodulator <NUM> may separate the received broadcast signal into an image signal, an audio signal, and a data signal related to a broadcast program, and restore the separated image signal, audio signal, and data signal to a format capable of being output.

The network interface <NUM> may provide an interface for connecting the display device <NUM> to a wired/wireless network including an Internet network. The network interface <NUM> may transmit or receive data to or from other users or other electronic devices through a connected network or another network linked to the connected network.

The network interface <NUM> may access a predetermined web page through the connected network or the other network linked to the connected network. That is, it is possible to access a predetermined web page through a network, and transmit or receive data to or from a corresponding server.

Then, the network interface <NUM> may receive contents or data provided from a content provider or a network operator. That is, the network interface <NUM> can receive content such as movies, advertisements, games, video on demands (VODs), and broadcast signals, which are provided from a content provider or a network provider, through network and information relating thereto.

In addition, the network interface <NUM> may receive update information and update files of firmware provided by the network operator, and may transmit data to an Internet or content provider or a network operator.

The network interface <NUM> may select and receive a desired application from among applications that are open to the public through a network.

The external device interface <NUM> may receive an application or a list of applications in an external device adjacent thereto, and transmit the same to the controller <NUM> or the memory <NUM>.

The external device interface <NUM> may provide a connection path between the display device <NUM> and an external device. The external device interface <NUM> may receive one or more of images and audio output from an external device connected to the display device <NUM> in a wired or wireless manner, and transmit the same to the controller <NUM>. The external device interface <NUM> may include a plurality of external input terminals. The plurality of external input terminals may include an RGB terminal, one or more High Definition Multimedia Interface (HDMI) terminals, and a component terminal.

The image signal of the external device input through the external device interface <NUM> may be output through the display <NUM>. The audio signal of the external device input through the external device interface <NUM> may be output through the speaker <NUM>.

The external device connectable to the external device interface <NUM> may be any one of a set-top box, a Blu-ray player, a DVD player, a game machine, a sound bar, a smartphone, a PC, a USB memory, and a home theater, but this is an example.

In addition, some content data stored in the display device <NUM> may be transmitted to another user registered in advance in the display device <NUM> or a selected user or a selected electronic device among other users or other electronic devices.

The memory <NUM> may store programs for signal processing and control of the controller <NUM>, and may store images, audio, or data signals, which have been subjected to signal-processed.

In addition, the memory <NUM> may perform a function for temporarily storing images, audio, or data signals input from an external device interface <NUM> or the network interface <NUM>, and store information on a predetermined image through a channel storage function.

The memory <NUM> may store an application or a list of applications input from the external device interface <NUM> or the network interface <NUM>.

The display device <NUM> may play back a content file (a moving image file, a still image file, a music file, a document file, an application file, or the like) stored in the memory <NUM> and provide the same to the user.

The user input interface <NUM> may transmit a signal input by the user to the controller <NUM> or a signal from the controller <NUM> to the user. For example, the user input interface <NUM> may receive and process a control signal such as power on/off, channel selection, screen settings, and the like from the remote control device <NUM> in accordance with various communication methods, such as a Bluetooth communication method, a Ultra Wideband (UWB) communication method, a ZigBee communication method, a Radio Frequency (RF) communication method, or an infrared (IR) communication method or may perform processing to transmit the control signal from the controller <NUM> to the remote control device <NUM>.

In addition, the user input interface <NUM> may transmit a control signal input from a local key (not shown) such as a power key, a channel key, a volume key, and a setting value to the controller <NUM>.

The image signal image-processed by the controller <NUM> may be input to the display <NUM> and displayed as an image corresponding to a corresponding image signal. Also, the image signal image-processed by the controller <NUM> may be input to an external output device through the external device interface <NUM>.

The audio signal processed by the controller <NUM> may be output to the speaker <NUM>. Also, the audio signal processed by the controller <NUM> may be input to the external output device through the external device interface <NUM>.

In addition, the controller <NUM> may control the overall operation of the display device <NUM>.

In addition, the controller <NUM> may control the display device <NUM> by a user command input through the user input interface <NUM> or an internal program and connect to a network to download an application a list of applications or applications desired by the user to the display device <NUM>.

The controller <NUM> may allow the channel information or the like selected by the user to be output through the display <NUM> or the speaker <NUM> along with the processed image or audio signal.

In addition, the controller <NUM> may output an image signal or an audio signal through the display <NUM> or the speaker <NUM>, according to a command for playing back an image of an external device through the user input interface <NUM>, the image signal or the audio signal being input from an external device, for example, a camera or a camcorder, through the external device interface <NUM>.

Meanwhile, the controller <NUM> may allow the display <NUM> to display an image, for example, allow a broadcast image which is input through the tuner <NUM> or an external input image which is input through the external device interface <NUM>, an image which is input through the network interface or an image which is stored in the memory <NUM> to be displayed on the display <NUM>. In this case, an image being displayed on the display <NUM> may be a still image or a moving image, and may be a 2D image or a 3D image.

In addition, the controller <NUM> may allow content stored in the display device <NUM>, received broadcast content, or external input content input from the outside to be played back, and the content may have various forms such as a broadcast image, an external input image, an audio file, still images, accessed web screens, and document files.

The wireless communication interface <NUM> may communicate with an external device through wired or wireless communication. The wireless communication interface <NUM> may perform short range communication with an external device. To this end, the wireless communication interface <NUM> may support short range communication using at least one of Bluetooth™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), UWB, ZigBee, Near Field Communication (NFC), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies. The wireless communication interface <NUM> may support wireless communication between the display device <NUM> and a wireless communication system, between the display device <NUM> and another display device <NUM>, or between the display device <NUM> and a network in which the display device <NUM> (or an external server) is located through wireless area networks. The wireless area networks may be wireless personal area networks.

Here, the another display device <NUM> may be a wearable device (e.g., a smartwatch, smart glasses or a head mounted display (HMD), a mobile terminal such as a smart phone, which is able to exchange data (or interwork) with the display device <NUM> according to the present disclosure. The wireless communication interface <NUM> may detect (or recognize) a wearable device capable of communication around the display device <NUM>.

Furthermore, when the detected wearable device is an authenticated device to communicate with the display device <NUM> according to the present disclosure, the controller <NUM> may transmit at least a portion of data processed by the display device <NUM> to the wearable device through the wireless communication interface <NUM>. Therefore, a user of the wearable device may use data processed by the display device <NUM> through the wearable device.

The microphone <NUM> may acquire audio. The microphone <NUM> may include at least one microphone (not shown), and may acquire audio around the display device <NUM> through the microphone (not shown).

The display <NUM> may convert image signals, data signals, and OSD signals processed by the controller <NUM>, or image signals or data signals received from the external device interface <NUM> into R, G, and B signals, and generate drive signals.

Meanwhile, since the display device <NUM> shown in <FIG> is only an embodiment of the present disclosure, some of the illustrated components may be integrated, added, or omitted depending on the specification of the display device <NUM> that is actually implemented.

That is, two or more components may be combined into one component, or one component may be divided into two or more components as necessary. In addition, a function performed in each block is for describing an embodiment of the present disclosure, and its specific operation or device does not limit the scope of the present disclosure.

Unlike the display device <NUM> shown in <FIG>, the display device <NUM> may receive an image through the network interface <NUM> or the external device interface <NUM> without a tuner <NUM> and a demodulator <NUM> and play back the same.

For example, the display device <NUM> may be divided into an image processing device, such as a set-top box, for receiving broadcast signals or content according to various network services, and a content playback device that plays back content input from the image processing device.

In this case, an operation method of the display device according to an embodiment of the present disclosure will be described below may be implemented by not only the display device <NUM> as described with reference to <FIG> and but also one of an image processing device such as the separated set-top box and a content playback device including the display <NUM> and the speaker <NUM>.

The speaker <NUM> may receive a signal audio-processed by the controller <NUM> and output the same with audio.

The power supply circuit <NUM> may supply corresponding power to the display device <NUM>. Particularly, power may be supplied to the controller <NUM> that may be implemented in the form of a system on chip (SOC), the display <NUM> for image display, and the speaker <NUM> for audio output.

Specifically, the power supply circuit <NUM> may include a converter that converts AC power into DC power, and a dc/dc converter that converts a level of DC power.

Next, a remote control device according to an embodiment of the present disclosure will be described with reference to <FIG>.

<FIG> is a block diagram of a remote control device according to an embodiment, not encompassed by the wording of the claims but useful for understanding context of the invention, and <FIG> illustrates an actual configuration example of a remote control device <NUM> according to an embodiment, not encompassed by the wording of the claims but useful for understanding context of the invention.

First, referring to <FIG>, the remote control device <NUM> may include a fingerprint reader <NUM>, a wireless communication circuit <NUM>, a user input interface <NUM>, a sensor <NUM>, an output interface <NUM>, a power supply circuit <NUM>, a memory <NUM>, a controller <NUM>, and a microphone <NUM>.

Referring to <FIG>, the wireless communication circuit <NUM> may transmit and receive signals to and from any one of display devices according to embodiments of the present disclosure described above.

The remote control device <NUM> may include an RF circuit <NUM> capable of transmitting and receiving signals to and from the display device <NUM> according to the RF communication standard, and an IR circuit <NUM> capable of transmitting and receiving signals to and from the display device <NUM> according to the IR communication standard. In addition, the remote control device <NUM> may include a Bluetooth circuit <NUM> capable of transmitting and receiving signals to and from the display device <NUM> according to the Bluetooth communication standard. In addition, the remote control device <NUM> may include an NFC circuit <NUM> capable of transmitting and receiving signals to and from the display device <NUM> according to the NFC communication standard, and a WLAN circuit <NUM> capable of transmitting and receiving signals to and from the display device <NUM> according to the WLAN communication standard.

In addition, the remote control device <NUM> may transmit a signal containing information on the movement of the remote control device <NUM> to the display device <NUM> through the wireless communication circuit <NUM>.

In addition, the remote control device <NUM> may receive a signal transmitted by the display device <NUM> through the RF circuit <NUM>, and transmit a command regarding power on/off, channel change, volume adjustment, or the like to the display device <NUM> through the IR circuit <NUM> as necessary.

The user input interface <NUM> may include a keypad, a button, a touch pad, a touch screen, or the like. The user may input a command related to the display device <NUM> to the remote control device <NUM> by operating the user input interface <NUM>. When the user input interface <NUM> includes a hard key button, the user may input a command related to the display device <NUM> to the remote control device <NUM> through a push operation of the hard key button. Details will be described with reference to <FIG>.

Referring to <FIG>, the remote control device <NUM> may include a plurality of buttons. The plurality of buttons may include a fingerprint recognition button <NUM>, a power button <NUM>, a home button <NUM>, a live button <NUM>, an external input button <NUM>, a volume control button <NUM>, a voice recognition button <NUM>, a channel change button <NUM>, an OK button <NUM>, and a back-play button <NUM>.

The fingerprint recognition button <NUM> may be a button for recognizing a user's fingerprint. In one embodiment, the fingerprint recognition button <NUM> may enable a push operation, and thus may receive a push operation and a fingerprint recognition operation. The power button <NUM> may be a button for turning on/off the power of the display device <NUM>. The home button <NUM> may be a button for moving to the home screen of the display device <NUM>. The live button <NUM> may be a button for displaying a real-time broadcast program. The external input button <NUM> may be a button for receiving an external input connected to the display device <NUM>. The volume control button <NUM> may be a button for adjusting the level of the volume output by the display device <NUM>. The voice recognition button <NUM> may be a button for receiving a user's voice and recognizing the received voice. The channel change button <NUM> may be a button for receiving a broadcast signal of a specific broadcast channel. The OK button <NUM> may be a button for selecting a specific function, and the back-play button <NUM> may be a button for returning to a previous screen.

A description will be given referring again to <FIG>.

When the user input interface <NUM> includes a touch screen, the user may input a command related to the display device <NUM> to the remote control device <NUM> by touching a soft key of the touch screen. In addition, the user input interface <NUM> may include various types of input means that may be operated by a user, such as a scroll key or a jog key, and the present embodiment does not limit the scope of the present disclosure.

The sensor <NUM> may include a gyro sensor <NUM> or an acceleration sensor <NUM>, and the gyro sensor <NUM> may sense information regarding the movement of the remote control device <NUM>.

For example, the gyro sensor <NUM> may sense information about the operation of the remote control device <NUM> based on the x, y, and z axes, and the acceleration sensor <NUM> may sense information about the moving speed of the remote control device <NUM>. Meanwhile, the remote control device <NUM> may further include a distance measuring sensor to sense the distance between the display device <NUM> and the display <NUM>.

The output interface <NUM> may output an image or audio signal corresponding to the operation of the user input interface <NUM> or a signal transmitted from the display device <NUM>. The user may recognize whether the user input interface <NUM> is operated or whether the display device <NUM> is controlled through the output interface <NUM>.

For example, the output interface <NUM> may include an LED <NUM> that emits light, a vibrator <NUM> that generates vibration, a speaker <NUM> that outputs sound, or a display <NUM> that outputs an image when the user input interface <NUM> is operated or a signal is transmitted and received to and from the display device <NUM> through the wireless communication circuit <NUM>.

In addition, the power supply circuit <NUM> may supply power to the remote control device <NUM>, and stop power supply when the remote control device <NUM> has not moved for a predetermined time to reduce power consumption. The power supply circuit <NUM> may restart power supply when a predetermined key provided in the remote control device <NUM> is operated.

The memory <NUM> may store various types of programs and application data required for control or operation of the remote control device <NUM>.

When the remote control device <NUM> transmits and receives signals wirelessly through the display device <NUM> and the RF circuit <NUM>, the remote control device <NUM> and the display device <NUM> transmit and receive signals through a predetermined frequency band.

The controller <NUM> of the remote control device <NUM> may store and refer to information on a frequency band capable of wirelessly transmitting and receiving signals to and from the display device <NUM> paired with the remote control device <NUM> in the memory <NUM>.

The controller <NUM> may control all matters related to the control of the remote control device <NUM>. The controller <NUM> may transmit a signal corresponding to a predetermined key operation of the user input interface <NUM> or a signal corresponding to the movement of the remote control device <NUM> sensed by the sensor <NUM> through the wireless communication circuit <NUM>.

Also, the microphone <NUM> of the remote control device <NUM> may obtain a speech.

A plurality of microphones <NUM> may be provided.

Next, a description will be given referring to <FIG>.

<FIG> illustrates an example of using a remote control device according to an embodiment, not encompassed by the wording of the claims but useful for understanding the context of the invention.

In <FIG>, it is illustrated that a pointer <NUM> corresponding to the remote control device <NUM> is displayed on the display <NUM>.

The user may move or rotate the remote control device <NUM> up, down, left and right. The pointer <NUM> displayed on the display <NUM> of the display device <NUM> may correspond to the movement of the remote control device <NUM>. As shown in the drawings, the pointer <NUM> is moved and displayed according to movement of the remote control device <NUM> in a 3D space, so the remote control device <NUM> may be called a space remote control device.

In <FIG>, it is illustrated that that when the user moves the remote control device <NUM> to the left, the pointer <NUM> displayed on the display <NUM> of the display device <NUM> moves to the left correspondingly.

Information on the movement of the remote control device <NUM> detected through a sensor of the remote control device <NUM> is transmitted to the display device <NUM>. The display device <NUM> may calculate the coordinates of the pointer <NUM> based on information on the movement of the remote control device <NUM>. The display device <NUM> may display the pointer <NUM> to correspond to the calculated coordinates.

In <FIG>, it is illustrated that a user moves the remote control device <NUM> away from the display <NUM> while pressing a specific button in the remote control device <NUM>. Accordingly, a selected area in the display <NUM> corresponding to the pointer <NUM> may be zoomed in and displayed enlarged.

Conversely, when the user moves the remote control device <NUM> to be close to the display <NUM>, the selected area in the display <NUM> corresponding to the pointer <NUM> may be zoomed out and displayed reduced.

On the other hand, when the remote control device <NUM> moves away from the display <NUM>, the selected area may be zoomed out, and when the remote control device <NUM> moves to be close to the display <NUM>, the selected area may be zoomed in.

Also, in a state in which a specific button in the remote control device <NUM> is being pressed, recognition of up, down, left, or right movements may be excluded. That is, when the remote control device <NUM> moves away from or close to the display <NUM>, the up, down, left, or right movements are not recognized, and the forward and backward movements may be recognized. In a state in which a specific button in the remote control device <NUM> is not being pressed, the pointer <NUM> moves according to the up, down, left, or right movements of the remote control device <NUM>.

Meanwhile, the movement speed or the movement direction of the pointer <NUM> may correspond to the movement speed or the movement direction of the remote control device <NUM>.

Meanwhile, in the present specification, a pointer refers to an object displayed on the display <NUM> in response to an operation of the remote control device <NUM>. Accordingly, objects of various shapes other than the arrow shape shown in the drawings are possible as the pointer <NUM>. For example, the object may be a concept including a dot, a cursor, a prompt, a thick outline, and the like. In addition, the pointer <NUM> may be displayed corresponding to any one point among points on a horizontal axis and a vertical axis on the display <NUM>, and may also be displayed corresponding to a plurality of points such as a line and a surface.

5A and 5B are diagrams for describing a horizontal mode and a vertical mode of a stand-type display device according to an embodiment, not encompassed by the wording of the claims but useful for understanding the context of the invention.

Referring to FIGS. 5A and 5B, a stand-type display device <NUM> is illustrated.

A shaft <NUM> and a stand base <NUM> may be connected to the display device <NUM>.

The shaft <NUM> may connect the display device <NUM> and the stand base <NUM> to each other. The shaft <NUM> may extend vertically.

The lower end of the shaft <NUM> may be connected to the edges of the stand base <NUM>.

The lower end of the shaft <NUM> may be rotatably connected to the edges of the stand base <NUM>.

The display device <NUM> and the shaft <NUM> may rotate about a vertical axis with respect to the stand base <NUM>.

An upper portion of the shaft <NUM> may be connected to the rear surface of the display device <NUM>.

The stand base <NUM> may serve to support the display device <NUM>.

The display device <NUM> may be configured to include the shaft <NUM> and the stand base <NUM>.

The display device <NUM> may rotate around a point where the upper portion of the shaft <NUM> and the rear surface of the display <NUM> contact each other.

5A illustrates that the display <NUM> operates in a landscape mode in which the horizontal length is greater than the vertical length, and FIG. 5B illustrates that the display <NUM> operates in a portrait mode in which the vertical length is greater than the horizontal length.

A user may move while holding a stand-type display device. That is, the stand-type display device has improved mobility, unlike a fixed device, so that a user is not limited by an arrangement position.

Next, various embodiments of processing a voice signal in the display device <NUM> will be described.

Hereinafter, the term "voice signal" may indicate a signal including voice data of a speaker (user) making a request to the display device <NUM>.

In particular, in the present disclosure, a case in which pieces of voice data of at least two or more speakers (for example, referred to as 'multi-speakers' for convenience of description) is included in the voice signal is described as an example, but the present disclosure is not limited thereto.

When requests, that is, pieces of voice data are simultaneously received from multi-speakers, or received during a short time within a predetermined time range, the display device <NUM> may be configured to process the pieces of voice data as a single voice signal. In this case, the predetermined time range may indicate a time period, for example, from a point in time when the pre-input voice data is received to a point in time when the processing of the pre-input voice data is not yet performed in the display device <NUM>. The voice signal may include pieces of voice data for at least two different requests from a single speaker rather than multi-speakers. Meanwhile, the voice data may include various requests such as a request for content or an application, a request for search or information provision, and the like.

When a voice signal including the pieces of voice data of the multi-speakers is input, the display device <NUM> may be configured to identify each multi-speaker from the voice signal, identify the voice data of each identified multi-speaker, and perform operation based on the current operating state of the display device <NUM> and the identified speaker or voice data.

<FIG> is a block diagram illustrating a configuration of a display device <NUM> according to an embodiment of the invention.

<FIG> is a diagram illustrating a process of processing voice data of multi-speakers according to an embodiment of the invention.

The display device <NUM> includes a memory and a processor <NUM> communicating with the memory.

The processor <NUM> is configured to separate input voice data into pieces of voice data for speakers, and process the voice data differently according to the current playback mode of the display device <NUM>. when the current playback mode of the display device <NUM> is a multi-view mode, the processor <NUM> is configured to perform control such that the content according to the separated voice data of each multi-speaker is output to each multi-view screen area.

Meanwhile, the processor <NUM> may be configured to communicate with an internal or external DB, that is, a first DB <NUM> and a second DB <NUM> to perform data communication such as collecting, recording/storing, and extracting/reading data.

Referring to <FIG>, the processor <NUM> may include a communication unit <NUM>, a data receiving unit <NUM>, a speaker recognition/processing unit <NUM>, includes a simultaneous processing determination unit <NUM>, may include further a multi-action processing unit <NUM>, a speaker data processing unit <NUM> and a control unit <NUM>. The processor <NUM> may be configured to further include one or more other components in addition to the components shown in <FIG>, or vice versa. Some of the components of the processor <NUM> shown in <FIG> may be implemented in a module form with other component(s), or vice versa. For example, at least two of the speaker recognition/processing unit <NUM>, the simultaneous processing determination unit <NUM>, the multi-action processing unit <NUM>, and the speaker data processing unit <NUM> are modularized and exist individually or may be included in the control unit <NUM>.

The communication unit <NUM> may be configured to support a communication interface environment with other components of the display device <NUM> or an external device (including a server).

The data receiving unit <NUM> may be configured to receive (or acquire) data from another component or an external device through the communication unit <NUM>. In this case, the received data may include a voice signal including the voice data of the multi-speakers.

The speaker recognition/processing unit <NUM> may be configured to recognize multi-speakers in the voice signal received through the data receiving unit <NUM>, identify each speaker in the recognized multi-speakers, separate the identified voice data of each speaker from the voice signal, and process the voice data for example, as shown in <FIG>. At least a part of the process of recognizing speakers and performing processing may be performed by an external device, for example, a server.

Referring to <FIG>, when a voice signal including "Show me the weather" (first voice data by speaker A) and "Run YouTube" (second voice data by speaker B) is received, the display device <NUM> (or the server) may be configured to recognize each multi-speaker and identify and process the voice data of the recognized each speaker.

Referring to <FIG>, the speaker recognition/processing unit <NUM> in the processor <NUM> of the display device <NUM> may include a speaker recognition feature extraction module <NUM>, a time-stamp extraction module <NUM>, it includes a voice data separation module <NUM>, it may further include a first Speech to Text (STT) processing module <NUM> and a second STT processing module <NUM>. The speaker recognition/processing unit <NUM> may be configured to further include one or more components differently from the configuration shown in <FIG> or vice versa. In addition, at least two or more of the speaker recognition feature extraction module <NUM>, the time-stamp extraction module <NUM>, the voice data separation module <NUM>, the first STT processing module <NUM> and the second STT processing modules <NUM>, which constitute the speaker recognition/processing unit <NUM>, may be implemented as one module or vice versa.

The speaker recognition feature extraction module <NUM> may be configured to extract features for speaker recognition from a speaker recognition feature DB <NUM>. In this case, the speaker recognition feature DB <NUM> may be included in the display device <NUM> or the speaker recognition/processing unit <NUM>, but may be located remotely unlike the configuration illustrated in <FIG>. For example, the speaker recognition feature DB <NUM> located remotely (e.g., at the server) may provide related (speaker) data according to the request of the speaker recognition/processing unit <NUM>.

The time-stamp extraction module <NUM> may extract time-stamp information for a voice data section (e.g., a start section and an end section) corresponding to the extracted speaker recognition feature.

The voice data separation module <NUM> is configured to perform separation into pieces of voice data for speakers, for example, the first voice data ("Show me the weather") of speaker A and the second voice data ("run YouTube") of speaker B, based on the time-stamp information.

The STT processing unit may be configured to perform STT processing on the separated voice data.

Referring to <FIG>, the first STT processing module <NUM> may be configured to perform STT processing on the separated first voice data of the speaker A, and the second STT processing module <NUM> may be configured to perform STT processing on the separated second voice data of the speaker B.

The voice data of each speaker that has been separated as described above may be subjected to natural language processing (NLP) in an NLP module <NUM>.

At least one of the STT processing unit and the NLP module <NUM> may be included in the display device <NUM> or the server.

On the other hand, the STT and NLP processes or algorithms are referred to the known technology, and a separate detailed description thereof is omitted herein.

The simultaneous processing determination unit <NUM> is configured to determine whether pieces of voice data of multi-speakers are able to be simultaneously processed on the display device <NUM>. The simultaneous processing determination unit <NUM> may be configured to perform a function when the playback mode of the display device <NUM> is a single-view mode rather than a multi-view mode, but the present disclosure is not limited thereto.

The simultaneous processing determination unit <NUM> is configured to determine whether simultaneous output is possible in consideration of the relationship between the voice data of each multi-speaker and content currently being played. In this case, the simultaneous processing determination unit <NUM> is configured to perform a function of determining whether simultaneous output is possible even when the playback mode of the display device <NUM> is a multi-view mode rather than a single-view mode. That is, for each multi-view screen area, the simultaneous processing determination unit <NUM> is configured to determine whether content currently being output in a corresponding area and content according to the request (command) corresponding to the voice data of each multi-speaker are able to be output simultaneously or need to be replaced with each other.

The multi-action processing unit <NUM> may be configured to perform multi-action processing such that commands according to pieces of voice data of multi-speakers are processed together on the display device <NUM>.

The speaker data processing unit <NUM> may be configured to process actual data according to the pieces of voice data of the multi-speakers.

The control unit <NUM> may be configured to control the overall operation of the processor <NUM>, and in particular, perform control operation according to various situations for processing a voice signal including the pieces of voice data of multi-speakers.

The first DB <NUM> and the second DB <NUM> may be configured to perform operation of recording/storing or extracting/reading data, or the like through data communication with the processor <NUM>. Any one of the first DB <NUM> and the second DB <NUM> may correspond to or include the speaker recognition feature DB <NUM> of <FIG>.

<FIG> and <FIG> are diagrams for a method of processing a voice signal including data of multi-speakers according to an embodiment of the invention. <FIG> are diagrams for a method of processing a voice signal including data of multi-speakers, not encompassed by the wording of the claims,.

Hereinafter, for convenience of description, multi-speakers including two speakers will be described as an example, but the present disclosure is not limited thereto.

As used herein, the term 'content' may include any type of data capable of being played on a display, such as a broadcast program, an application, or an external input, unless otherwise specified.

The display device <NUM> is configured to receive voice data, separate the received voice data into pieces of voice data for speakers, and when the current playback mode is a multi-view mode, perform control such that content according to each piece of voice data, which has been separated for each speaker is output to each multi-view screen area.

The display device <NUM> may be configured to extract speaker recognition feature information from the DB. Specifically, the display device <NUM> may be configured to extract time-stamp information on a part corresponding to the extracted speaker recognition feature information from the received voice data, and separate the voice data into pieces of voice data for speakers based on the extracted time-stamp information, thus enabling the received voice data to be separated into pieces of voice data for speakers.

The display device <NUM> may be further configured to perform STT processing on the voice data separated for each speaker based on the extracted time-stamp information, and perform NLP on the voice data of each speaker which has been subjected to STT processing to enable the received voice data to be separated into the pieces of voice data for speakers.

The display device <NUM> may be configured to determine a multi-view screen area corresponding to each voice data separated for each speaker and perform mapping.

The display device <NUM> is configured to extract content information currently being played on each determined multi-view screen area.

The display device <NUM> is configured to compare content information currently being played on each extracted multi-view screen area with content according to voice data mapped to a corresponding multi-view screen area, and determine whether simultaneous output is possible.

When an operation command (content) according to the voice data mapped to the multi-view screen area is able to be output simultaneously as a result of determining whether simultaneous output is possible, the display device <NUM> is configured to output data according to the operation command on one area of the content screen being currently played in the corresponding screen area.

When an operation command (content) according to the voice data mapped to the multi-view screen area is unable to be output simultaneously as a result of determining whether simultaneous output is possible, the display device <NUM> may be configured to output data according to the operation command onto a corresponding screen area instead of content currently being played.

The display device <NUM> may be configured to determine whether a request according to each voice data separated for each speaker is a request incapable of being processed simultaneously, when it is determined that the request is a request incapable of being simultaneously being processed as a result of the determination, determine a main speaker, and activate a request according to voice data corresponding to the determined main speaker from among pieces of voice data separated for speakers.

The display device <NUM> may be configured to extract content information currently being played on each multi-view screen area, and determine a multi-view screen on which a request according to voice data corresponding to the activated main speaker it to be output, based on the extracted content information of each multi-view screen area. The display device <NUM> may be configured to determine whether simultaneous output is possible based on the content information of a multi-view screen on which the request according to the voice data corresponding to the main speaker is to be output and, when simultaneous output is possible, output the content according to the request onto one area on the content of the multi-view screen.

The display device <NUM> may be configured to upload voice data, separated for each speaker as described above, to a server, and download an NLP response, generated by performing NLP on the uploaded voice data, from the server.

The controlling the operation of the display device <NUM> in a server may comprise receiving voice data separated for each speaker from received voice data, performing STT processing and NLP on each voice data for each speaker, and generate and transmit information on each voice data which has been subjected to NLP, that is, an NLP response to the display device <NUM>. In this case, when the current playback mode of the display device <NUM> is recognized as the multi-view mode, the server may perform control such that content corresponding to the generated NLP response is output to each multi-view screen area.

Referring to <FIG> when a voice input is received (S101), the display device <NUM> may be configured to determine whether there are multi-speakers (S103). In this case, determination as to whether pieces of voice data of multi-speakers are included in the received input voice may be performed according to, for example, the related description of <FIG> and/or <NUM> as described above.

When it is determined that the input voice includes the pieces of voice data of multi-speakers as a result of the determination in step S103, the display device <NUM> may be configured to determine whether the current playback mode of the display device <NUM> is the multi-view mode (S105).

When the current playback mode of the display device <NUM> is the multi-view mode as a result of the determination in step S105, the display device <NUM> may be configured to perform STT processing on each speaker's voice data and transmit the data that has been subjected to STT processing to the NLP (S107).

After transmitting the data to the NLP in step S107, the display device <NUM> may be configured to receive a result of STT processing, that is, an NLP response from the NLP, and perform control to output the NLP response to each multi-view screen area as shown in <FIG> (S109).

Referring to <FIG> and <FIG> , weather information is provided on a content playback screen in a first multi-view screen area <NUM>, and an execution screen of YouTube application may be provided on a second multi-view screen area <NUM> instead of previously-provided content.

In <FIG>, the display device <NUM> may be configured to omit step S105 and, when there are multi-speakers, automatically execute the multi-view mode as shown in <FIG>, and provide outputs according to the voice data of the first speaker and the voice data of the second speaker on the multi-view screen areas.

In <FIG>, when the current playback mode is the single-view mode rather than the multi-view mode as a result of the determination in step S105, the display device <NUM> may be configured to perform STT processing on the voice data of a specific speaker and transmit a result of STT processing to the NLP (S111).

Thereafter, the display device <NUM> may be configured to receive the NLP response and perform an operation according to the received NLP response through a screen as shown in <FIG> (S113).

Referring to of <FIG> , in step S111, a specific speaker may be regarded as, for example, a main speaker. The main speaker may be determined to be one of the multi-speakers, for example, a speaker currently logged-in to the display device <NUM>, a speaker having a higher relevance to the content currently being played on the screen, a speaker having a relatively higher preset speaker priority, or the like, according to various methods. For example, in <FIG>, it can be seen that weather information is output on the content currently being played. In this case, for example, since weather information requested by the main speaker is content capable of being processed or output simultaneously with the content currently being played on the display device <NUM>, the weather information has been provided on the content being currently played, as shown in <FIG>. However, unlike the above description, when the operation (e.g., content) requested by the main speaker is incapable of being simultaneously processed or output simultaneously with the content currently being played on the display device <NUM>, the content being currently played may be replaced with the operation requested by the main speaker unlike <FIG>. In this case, the content that had been played and is being replaced may be, for example, in a standby state in a background while playback of the content is suspended.

When the operation (content) included in the NLP response is incapable of being simultaneously processed or output simultaneously with the content being currently played in the display device <NUM> and the content being currently played is needed to be replaced eventually, a speaker who has requested the operation (content) capable of being output together with the content being currently played or the speaker's voice data may be processed with a relatively higher priority.

At the time of processing the voice input of multi-speakers, the display device <NUM> may be configured to obtain state information of a resource, and select a specific speaker as the main speaker based on the state information of the resource with respect to the voice input of the multi-speakers.

The contents mentioned in each of the above-described embodiments function as weights and are given weights according to the settings, so that the display device <NUM> may be configured to use the weight values or the sum thereof in various determination processes described in this specification.

Next, referring to <FIG>, when a voice input is received (S201), the display device <NUM> may be configured to determine whether the current playback mode is the multi-view mode (S203).

When the current playback mode is the multi-view mode as a result of the determination in step S203, the display device <NUM> may be configured to determine whether the voice data of multi-speakers is included in the received voice input (S205).

When it is determined in step S205 that the received voice input is a single-speaker input, the display device <NUM> may be configured to select or determine a multi-view screen area to which the single-speaker input is to be output (S207).

The display device <NUM> may be configured to process content according to the single-speaker input (S211), and output the processed content to the multi-view screen area selected or determined through the step S207 (S213).

In relation to step S207, the display device <NUM> may be configured to refer to at least one of information about a speaker linked (or mapped) to each multi-view screen area and content information being played in each multi-view screen area. For example, when the multi-view screen areas are respectively assigned to different users, and speakers linked to the multi-view screen areas are different, it is preferable to output the processed content to the multi-view screen area assigned to a user matching the speaker of the single-speaker input.

Referring to <FIG>, in a case where a first multi-view screen area <NUM> is assigned to a first speaker and a second multi-view screen area <NUM> is assigned to a second speaker, when a single-speaker input is the voice input of the first speaker, content may be provided to the first multi-view screen area <NUM> which is the area assigned to the first speaker, although there is no content being output on the second multi-view screen area <NUM>. When there is no content being output in the second multi-view screen area <NUM>, unlike the above case, content corresponding to the voice input of the first speaker may be played in the second multi-view screen area <NUM> until a content play request by the second speaker is received.

When it is determined in step S205 that the received voice input is a multi-speaker input, the display device <NUM> may be configured to select or determine each multi-view screen area <NUM> and <NUM> on which each multi-speaker input is to be output (S209).

The display device <NUM> is configured to process content according to each multi-speaker input (S211), and output the processed content in the multi-view screen area <NUM> or <NUM> selected or determined through the step S209 as shown in <FIG> (S213).

Referring to <FIG>, the display device <NUM> may be configured to determine an output screen area according to a corresponding speaker when the speaker linked to a multi-view screen area <NUM> or <NUM> matches a multi-speaker identified in the received voice input. However, when at least one speaker among the multi-speakers does not match the speaker linked to the multi-view screen area <NUM> or <NUM>, the display device <NUM> may be configured to select or determine a specific multi-view screen area for a corresponding multi-speaker by referring to information on content being played or information on speakers linked to the multi-view screen area <NUM> and <NUM>.

For example, it is assumed that the received voice input includes the voice input of multi-speakers, that is, the speaker <NUM> and speaker <NUM>, the voice input of the speaker <NUM> is a request for execution of application B, and the voice input of the speaker <NUM> is a request for weather information, the first multi-view screen area <NUM> is linked to the speaker <NUM>, application A is being executed, the second multi-view screen area <NUM> is linked to the speaker <NUM>, and a news application is being executed. In the above case, the voice input of the speaker <NUM> may be output to the first multi-view screen <NUM> as it is. However, in the case of the speaker <NUM>, since the second multi-view screen <NUM> is currently linked to the speaker <NUM>, a processing method thereof is required.

In this case, the display device <NUM> may be configured to determine whether the property of the content being provided on the second multi-view screen area <NUM> and an operation according to the voice data of the speaker <NUM> are able to be simultaneously output, and simultaneously provide content of two speakers is may be based on a result of the determination. For example, since the speaker <NUM> has requested weather information, and the second multi-view screen area <NUM> is related to a news application currently being output for the speaker <NUM>, simultaneous output is possible, so that the display device <NUM> may be configured to perform simultaneous output as shown in <FIG>.

On the other hand, based on a result of determination as to whether the property of the content being provided on the second multi-view screen area <NUM> and an operation according to the voice data of the speaker <NUM> are able to be simultaneously output, the display device <NUM> may need to provide any one content alone.

The display device <NUM> may be configured to refer to the speaker priority information of the speaker <NUM> and the speaker <NUM>.

The display device <NUM> may be configured to refer to the importance or content priority information of the content requested by the speaker <NUM> and the speaker <NUM>.

The display device <NUM> may be configured to select one of a plurality of speakers in consideration of both the speaker priority information and the importance of the content or the content priority information.

For example, the display device <NUM> may be configured to determine whether there is an additional input from the speaker <NUM> during a predetermined time period, and if not, select the speaker <NUM> instead of the speaker <NUM>.

The display device <NUM> may consider a relationship with the content being output in the first multi-view screen area <NUM>. For example, when audio output is requested for the content being output on the first multi-view screen area <NUM>, and audio output is also requested for the content being output on the second multi-view screen area <NUM>, it is necessary to mute one of the audio outputs or output the one audio output through another output means. In this case, it is preferable to output content that does not require audio output to the second multi-view screen area <NUM>, based on which the speaker <NUM> or the speaker <NUM> may be selected.

Next, referring to <FIG> and <FIG>, (not encompassed by the wording of the claims) the operation method may be changed depending on whether the voice inputs of the multi-speakers are able to be simultaneously processed (outputted) in the display device <NUM>. For convenience of description, the single-view mode is described as an example in <FIG> and <FIG>, but the present disclosure is not limited thereto.

Referring to <FIG>, when multi-speakers are recognized (S301), the display device <NUM> may be configured to determine whether simultaneous processing is possible (S303).

As a result of the determination in step S303, the display device <NUM> may be configured to perform different operations according to Cases <NUM> to <NUM>.

In Case <NUM> in which the voice inputs of the multi-speakers are simultaneously able to be processed, the display device <NUM> may perform multi-action processing (S305), and provide a result of the multi-action processing on a screen as shown in <FIG>.

For example, overlay applications that can be displayed on another full-size application may include a weather card application and a time card application. Accordingly, the display device <NUM> may be configured to determine whether simultaneous processing is possible based on whether an overlay card can be simultaneously executed on a full card. Referring to <FIG> as an example, the weather overlay card <NUM> may be output together on the YouTube full-size card <NUM>. In this case, it may be determined that simultaneous processing is possible.

When the voice inputs of the multi-speakers cannot be simultaneously processed, the display device <NUM> may be configured to again determine whether main speaker information is registered (S307).

Referring to <FIG>, Case <NUM> is a case in which simultaneous processing is impossible but a main speaker is registered, and Case <NUM> is a case in which simultaneous processing is impossible and a main speaker is not registered,.

When it is determined in step S307 that the main speaker is registered (Case <NUM>), the display device <NUM> may be configured to extract voice data of the registered main speaker (S309), and perform an operation according to the main speaker's voice data extracted main speaker as shown in <FIG> (S311).

On the other hand, when the main speaker is not registered as a result of the determination in step S307 (Case <NUM>), the display device <NUM> may be configured to automatically determine the main speaker (S313), extract the voice data of the the determined main speaker, and perform the corresponding operation as shown in <FIG> (S315).

'Whether simultaneous processing is possible' described in this specification may be determined in various ways based on the settings of a server or user or learning results, and various factors such as speaker information, content information, time or space information, and environment information may be reflected. For example, when the voice inputs of the multi-speakers are an execution request for a first video application and an execution request for a second video application, respectively, the display device <NUM> may be configured not to simultaneously process two video application execution requests due to resource issues or the like. Therefore, this case may be regarded as a case in which simultaneous processing is impossible.

Referring to <FIG>, the display device <NUM> may be configured to output the first video data of the first content of the first application to the first multi-view area <NUM> and play the eighth video data of the same content of the same application on the second multi-view area <NUM>.

In this case, when the same content of the same application or video data of a different series are assigned to the first multi-view area <NUM> and the second multi-view area <NUM> in the display device <NUM>, there may be interference with viewing of at least one speaker. Accordingly, in the above case, the display device <NUM> may be configured to determine a main speaker and output one content requested by the main speaker. That is, the display device <NUM> may be configured to determine that simultaneous processing is impossible in the above case.

On the other hand, in the case of <FIG>, the display device <NUM> may be configured to determine that simultaneous processing is impossible as described above even when pieces of content respectively output on the multi-view areas are same and applications executed to provide the content are different from each other.

Referring to <FIG>, the display device <NUM> may be configured to assign a first application to the first multi-view area <NUM> and assign a second application to the second multi-view area <NUM>. In this case, the display device <NUM> may be configured to determine whether simultaneous processing is possible or impossible. For example, when there is no particular setting, the display device <NUM> may be configured to determine whether the two applications can be simultaneously processed according to, for example, resource state information. Referring to <FIG>, for example, when video application execution requests are assigned to both multi-view areas <NUM> and <NUM> and resources are sufficient to process both video applications together, the display device <NUM> is able to perform simultaneous output. However, in this case, audio processing may be a problem, and when there are no settings for audio processing, the display device <NUM> may be configured to determine that simultaneous processing is impossible, determine a main speaker and perform simultaneous processing and output.

Referring to <FIG>, the display device <NUM> may be configured to allocate a first application to the first multi-view area <NUM> and allocate a second application to the second multi-view area <NUM>. In addition, the applications respectively allocated to the multi-view areas are an Internet application and a video application, and when there is no particular setting, the display device <NUM> may be configured to determine that simultaneous processing is possible.

Meanwhile, in <FIG>, the allocation to each multi-view screen may be performed based on the voice inputs of multi-speakers.

Referring to <FIG>, when a voice signal is received (S401), the display device <NUM> may be configured to determine whether multi-speaker voice data is included in the voice signal. According to a result of the determination, the display device <NUM> may be configured to perform separation on a voice signal in case of multi-speakers or process a voice signal of a single-speaker.

In this case, when the voice signal includes multi-speaker voice data, the display device <NUM> may be configured to determine a main speaker from among the multi-speakers (S403).

When the main speaker is identified through the step S403, the display device <NUM> may be configured to read routine information for the main speaker as shown in <FIG> (S405). The routine information is, for example, information previously stored in a memory, a DB, or the like, and may be information prepared based on a speaker and a wake-up word (or a trigger word). For example, information in which a speaker-wake-up word-routine is mapped is exemplified in <FIG>, but the present disclosure is not limited thereto.

Referring to <FIG>, the display device <NUM> may be configured to perform and store mapping on the routine of "Switching news channels - Watching weather" for wake-up word "good morning" and perform mapping on the routine of "Executing gallery - Watching tomorrow weather - Turning off TV in <NUM> minutes" for wake-up word "good night" with respect to speaker "A".

On the other hand, referring to <FIG>, the display device <NUM> may be configured to perform and store mapping on the routine of "playing YouTube music - Watching weather" for wake-up word "good morning" and perform and store mapping on the routine of "playing YouTube music - Turning off TV in <NUM> minutes" for wake-up word "good night" with respect to speaker "B".

The display device <NUM> may be configured to call at least one application corresponding to read routine information (S407) and output content by executing the application (S409).

As described above, when the main speaker is speaker "B" and the wake-up word is "good morning", the display device <NUM> may be configured to play YouTube music and then output weather information according to the read routine information.

When the current playback mode is a single-view mode, the display device <NUM> may be configured to call and execute two applications simultaneously as shown in <FIG>, and overlay and provide one application execution screen <NUM> on another application execution screen <NUM>. Although the display device <NUM> simultaneously calls and executes a plurality of applications included in the routine information in <FIG>, but the present disclosure is not limited thereto. For example, each application execution screen included in the routine information may be overlaid and output on an application screen that is being played on the display device <NUM> irrespective of the routine information. In this case, for example, the display device <NUM> is able to output the two applications at the same time, but in other cases, the applications included in the routine information may be executed instead of the previously executed application. In the latter case, as shown in <FIG>, the display device <NUM> may be configured to switch to the multi-view mode and perform simultaneous output. In this case, the display device <NUM> may be configured to execute one application according to the routine information, and when the execution of the corresponding application is terminated (or terminated after execution for a preset time), subsequently execute another application defined in the routine information. As described above, whether a plurality of mapped applications is simultaneously provided or sequentially provided may be determined according to, for example, an attribute of each application in the routine information, with reference to the routine information.

When the current playback mode is the multi-view mode, the display device <NUM> may be configured to simultaneously execute two applications and map the two applications to multi-view screens <NUM> and <NUM>, respectively, and then provide the multi-view screens, as shown in <FIG>. In this case, the allocation of the screen areas to the applications may be arbitrary, and may be determined according to the speaker's input or the speaker's current location after the screens are provided. Alternatively, when three or more applications are included in the routine information, any one of the two screen areas may be selected and the remaining application may be executed after the application executed in the corresponding screen area is terminated. For example, when the total playback time of the content on the first screen area is one hour and the total playback time of the content on the second screen area is <NUM> minutes, the display device <NUM> may be configured to allocate the second screen as the execution screen of the remaining application.

Step S403 in <FIG> described above may not be necessary. For example, the display device <NUM> may be configured to determine a main speaker when there are multi-speakers and the current playback mode is the single-view mode, and skip step S403 when the current playback mode is the multi-view mode.

Meanwhile, the routine information of <FIG> is an example, and is not limited thereto. For example, the wake-up word may not be considered. Current time information, content information, or the like may be considered instead of the wake-up word. The display device <NUM> may be configured to continuously update and store such routine information with reference to training through an AI learning model, feedback (e.g., motion, voice input, etc.) of each speaker, or the like.

Although category information is just defined for each speaker in <FIG>, specifically, which content or information of a corresponding category (or application) is to be output may be defined in more detail.

Referring to <FIG> , when the current playback mode is the multi-view mode, the display device <NUM> may be configured to allocate a first view area <NUM> to a user A and a second view area <NUM> to a user B.

The display device <NUM> may be configured to detect a user periodically or aperiodically, and when a specific user among the user A and the user B is identified as being away from the seat as a result of the detection, perform a process according to a result of the identification. The process may be differently performed depending on whether, for example, the user who is away from the seat is the main speaker.

For convenience of description, it is assumed that the main speaker in <FIG> is the user A.

When it is identified that the user A who is the main speaker is is away from the seat, the display device <NUM> may be configured to maintain the multi-view mode as before, even though the user B is continuously detected as shown in <FIG>. However, in this case, the playback of content for the user A, who is the main speaker, may be paused, or provided through a mirroring service, or the like. When the mirroring service is provided, the display device <NUM> may be configured to switch to the full screen for the user B as shown in <FIG>. On the other hand, when a time period during which the user A is away from the seat is equal to or longer than a preset time period, the display device <NUM> may be configured to switch to the full screen for the user B as shown in <FIG>.

On the other hand, when it is identified that the user B, who is not the main speaker, is away from the seat, the display device <NUM> may be configured to switch from the multi-view mode to the single-view mode, that is, the full screen mode, as shown in <FIG>. In this case, the content for the user B may be in a standby state in the background as the content for the user B is terminated or playback is stopped or paused.

In the above-described embodiments of the present disclosure, the wake-up word may be referred to in determining a main speaker based on a voice signal including pieces of voice data of multi-speakers.

For example, when a wake-up word such as "Hi LG" is included in the voice signal, the display device <NUM> may be configured to extract a time-stamp for the wake-up word and determine a corresponding speaker as the main speaker. In this case, the display device <NUM> may not consider the order (or time order) of the pieces of voice data of the multi-speakers.

On the other hand, in a case in which the pieces of voice data of the multi-speakers include all wake-up words, the display device <NUM> may be configured to extract each time-stamp as described above, select the speaker of voice data which is first input as the main speaker based on the time-stamp, and ignore the wake-up words and voice data of other speakers. However, when the display device <NUM> is in the multi-view mode, the wake-up word of voice data which is input later and the and voice data may be a target on which extraction and processing are to be performed. In the latter case, the display device <NUM> may be configured to extract and process the wake-up word of voice data which is input later and the voice data even when the display device <NUM> is not in the multi-view mode, but the display device <NUM> may be configured to perform output of the voice data when the playback according to the input data which is input previously has been stopped and completed.

When the wake-up words are respectively included in the pieces of voice data of the multi-speakers, the main speaker may be determined by referring to priority information of a command or request following each wake-up word regardless of the order thereof.

When the wake-up words are respectively included in the pieces of voice data of the multi-speakers, the main speaker may be determined according to priority information of each speaker or signal levels for the wake-up words.

When the wake-up words are respectively included in the pieces of voice data of the multi-speakers, the main speaker may be determined based on the relation between the command or request following each wake-up word and the currently played content of the display device <NUM>.

When the wake-up words are included all in the pieces of voice data of multi-speakers, the main speaker may be determined based on the current location of the display device <NUM>, information about a space to which the display device <NUM> belongs, time information, and the like. For example, when the display device <NUM> is located in a space called, for example, a kitchen, more weight is given to a specific user (e.g., a cook or a mother) to determine the main speaker. In the above case, space information to which the display device <NUM> belongs or time information is a factor for determining the main speaker despite the existence of a wake-up word, and the same or more weight (or priority) may be given.

<FIG> is a flowchart of a method of processing multi-speaker data according to an embodiment, not encompassed by the wording of the claims.

The display device <NUM> may be configured to receive a voice signal including voice data of multi-speakers (S501).

The display device <NUM> may be configured to identify content being played through a current screen (S503).

The display device <NUM> may be configured to determine the relevance between each of the multi-speakers and the content (S505).

As a result of the determination in step S505, the display device <NUM> may be configured to specify a speaker having higher relevance to the identified content as a main speaker (S507).

The display device <NUM> may be configured to perform an operation based on voice data of a speaker specified as the main speaker (S509).

Meanwhile, the display device <NUM> may be configured to determine whether the current playback mode of the display device <NUM> is a multi-view mode when the relevance between the identified content and the speaker is not significant as a result of the determination in step S505 (S507).

When the current playback mode is the multi-view mode as a result of the determination in step S507, the display device <NUM> may be configured to map multi-view screen areas to the pieces of voice data of the multi-speakers, and provide operations caused by the inputs of the speakers on the corresponding multi-view screen areas according to the mapping.

The illustrated order of operations of <FIG>, <FIG> is an example, and is not limited thereto. In other words, one or more steps in the flowchart may be performed concurrently or may be operated in a different order than shown. In addition, although shown as one step in <FIG>, <FIG>, the corresponding step may be divided into a plurality of steps and operated, or vice versa.

In addition, although not shown, one or more operations related to the present disclosure may be further performed.

The above description is merely illustrative of the technical idea of the present disclosure, and various modifications and variations may be made without departing from the essential characteristics of the present disclosure by those skilled in the art to which the present disclosure pertains.

According to at least one of various embodiments of the present disclosure, it is possible to separate voice data input from a plurality of speakers into pieces of voice data for speakers and perform processing on the pieces of voice data.

It is possible to separate voice data input from a plurality of speakers into pieces of voice data for speakers and adaptively perform processing on the pieces of voice data depending on situations including a playback mode.

It is possible to separate voice data input from a plurality of speakers into pieces of voice data for speakers and perform operation matching a user's intention to increase convenience of use and product satisfaction.

Claim 1:
A method for operating a display device (<NUM>) comprising:
receiving (S101) voice data;
separating (S103) the received voice data into pieces of voice data for speakers; and
performing control such that pieces of content according to the pieces of voice data, which have been separated for the speakers are respectively output on multi-view screen areas, when a current playback mode is a multi-view mode, characterised in that
the performing of the control includes:
extracting content information currently being played in each of the multi-view screen areas, and
determining whether simultaneous output is possible based on comparing the extracted content information currently being played on each of the multi-view screen areas with content according to voice data mapped to a corresponding multi-view screen area, and outputting the content, when simultaneous output is possible.