Patent Description:
Internet of things is based on the idea that everyday devices, not just computers and computer networks, can be readable, recognizable, locatable, addressable, and controllable via an loT communications network (e.g., an ad-hoc system or the Internet).

All the devices in the loT communications network environment leverages a voice-based interaction system to perform the one or more operations as intended by a user. The voice-based interaction system includes receiving a voice command (e.g., voice search command, speech, voice query, and/or voice search query etc.,) by the devices which are in vicinity thereto can facilitate user(s) with a response/service corresponding to the received voice command.

However, as in the loT environment with multiple devices (i.e., voice assistance) around, the user initiates a voice query and all the devices processes it and responds individually with zero coordination there between.

Patent publication <CIT> describes device selection for providing a response.

In accordance with an aspect of the disclosure, a method for managing voice-based interaction in an loT network system is provided, the method as set out in claim <NUM>.

In accordance with another aspect of the disclosure, an electronic device for managing voice-based interaction in an loT network system is provided, the electronic device as set out in claim <NUM>.

Accordingly, it should be apparent to those skilled in the art that the following description of various embodiments of the disclosure is provided for illustration purpose only and not for the purpose of limiting the disclosure as defined by the appended.

Herein, the term "or" as used herein, refers to a non-exclusive or, unless otherwise indicated. Accordingly, the examples should not be construed as limiting the scope of the claims.

Embodiments are described and illustrated in terms of blocks which carry out a described function or functions. These blocks, which may be referred to herein as managers, engines, controllers, units or modules or the like, are physically implemented by analog and/or digital circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hardwired circuits and the like, and may optionally be driven by firmware and software. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like. The circuits constituting a block may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware to perform some functions of the block and a processor to perform other functions of the block. Each block of the embodiments may be physically separated into two or more interacting and discrete blocks without departing from the scope of the claims. Likewise, the blocks of the embodiments may be physically combined into more complex blocks without departing from the scope of the claims.

The term "loT" and variations thereof, as used herein, refers to "internet of things". As such, the terms "loT" and "IOT" can be used interchangeably.

The term "zone", "location", "places" used herein can be used interchangeably without departing from the scope of the claims.

Continuous voice assistance: Here continuous voice assistance means when the user is moving across the different rooms (zones, location, etc.,), where single assistant (IoT) cannot hear the complete utterance of the user. All the assistants with multiple microphone channels, can address this when the user speaks complete utterance to the same assistant (without moving to another rooms).

<FIG> is an example of an Internet of things (loT) network system <NUM> in which an loT device individually processes multiple voice utterance received from a user(s) from different location, according to an embodiment of the disclosure.

Referring to <FIG>, consider an example scenario in which the user is moving freely across different zones (i.e., zone 1A and zone 1B) while conversation with each device 100a and 100b present therein. The conversation includes for e.g., voice utterane-<NUM> (book a movie ticket) heard by device 100a, and voice utterane-<NUM> (and also book a cab from my place to a cinema hall) heard by the device 100b. Hence, a user intended activity would be the voice utterance-<NUM> + voice utterance-<NUM>.

On the other hand, in order to provide the user intended response/service, the user needs to utter complete sentence in front of only one device (100a or 100b). The user cannot move naturally while having conversation with multiple devices (100a and 100b). If the voice is heard by multiple devices (100a and 100b), then both the devices 100a and the 100b will process and respond back simultaneously, which disrupt the user experience.

Further, even if the voice utterance is heard by both the devices 100a and the 100b, then both the devices 100a and the 100b will process the voice query associated with the voice utterance and respond back simultaneously, again disrupting the user experience.

Accordingly, embodiments herein provide a method and apparatus for managing voice-based interaction in an loT network system. The method includes identifying, by a voice assistance apparatus, a first voice utterance from a first IoT device in the loT network system. Further, the method includes identifying, by the voice assistance apparatus, at least one second voice utterance from at least one second IoT device in the loT network system. Further, the method includes determining, by the voice assistance apparatus, a voice command by combining the first voice utterance and the at least one second voice utterance. Furthermore, the method includes triggering, by the voice assistance apparatus, at least one IoT device in the loT network system to perform at least one action corresponding to the voice command.

Unlike related art methods and systems, the proposed method can be used to consolidate a portion of a voice utterance received at all the loT devices on a time basis (different zones).

For example, if the same voice utterance is heard at multiple loT devices, then a preference, to respond to the voice utterance, is provided to IoT device that heard best. Further, the proposed method can be used to identify the user intent based on a consolidated voice utterance (voice command) and process the voice command. Furthermore, the method includes sending the response to the best possible IoT device in a vicinity to the user. Thus, providing continuous and seamless experiences to the user.

<FIG> illustrates a high-level architecture of an loT network system <NUM>, according to an embodiment of the disclosure.

Referring to <FIG>, the electronic device 100a-100f (hereinafter collectively referred to as electronic device <NUM>) can be, for example, a smartphone, a cellular phone, a tablet, a phablet, a dual display device, a portable multimedia device, personal digital assistants (PDAs), a gaming controller, edge-to-edge displays, etc. Further, the electronic device <NUM> can be a wearable device such as, for example, a smart watch, a smart bracelet, a smart glass, or the like. In another embodiment the electronic device <NUM> can be loT device <NUM>.

The loT device <NUM> can be referred to any object (e.g., mobile devices, sensor, smart appliances such as refrigerators, TV, air conditioner (AC), toasters, ovens, microwaves, grills, rice cookers, juicers, freezers, dishwashers, dishes, hand tools, clothes washers, clothes dryers, water heaters, furnaces, air conditioners, thermostats, televisions, light, vacuum cleaners, sprinklers, electricity meters, gas meters, photo frames, watches, clocks, fans, speakers, etc.,) communicating information to one or more other objects through an interface <NUM> associated with each object respectively. The interface <NUM> such as, for example, an internet protocol (IP) address, wireless internet protocol, such as IEEE <NUM>, a Bluetooth identifier (ID), RFID, a near field communication (NFC) ID, etc. In other words, any object capable of accessing resources/services (e.g., authentication, providing assistance to the user, etc.,) in the loT network system <NUM> can simply be referred to the loT device <NUM>. Although, the loT devices <NUM> are shown to be communicating over the interface <NUM>, but it is not limited thereto, the and loT device <NUM> can also communicate over the direct wired connection (not shown) there between.

The loT device <NUM> is configured to communicate with a network <NUM> through an access point (AP) (not shown) over a physical communications interface or layer. The network <NUM> can be a mobile network to which the loT device <NUM> with a built-in modem capable can directly be connected. In another embodiment, the network <NUM> can be a fixed network through which the loT device <NUM> connects via a gateway, the fixed network can be able to aggregate the loT device(s) <NUM>.

Further, the loT network system <NUM> includes a voice assistance apparatus <NUM> configured to manage the voice-signal of the user received from the loT device(s) <NUM>. In an embodiment, the voice assistance apparatus <NUM> can be located at a server (not shown) and can remotely accessible through the network <NUM>, can be located in a cloud-based architecture, i.e., a cloud based distributed architecture, and a cloud based centralized architecture. In another embodiment, the voice assistance apparatus <NUM> can be located in an Ad-hoc architecture, i.e., an Ad-hoc centralized architecture, and Ad-hoc distributed architecture. Although not shown, the loT device <NUM> includes various sensors such as, for example, proximity sensors, optical, ambient light, temperature, pressure, Inertia, humidity, proximity, gesture, touch and fingerprint sensing applications. These sensors are based on a micro-electro-mechanical systems (MEMS) technology. Further, but not limited to, the loT device <NUM> also includes motion/velocity/displacement/position sensors.

<FIG> illustrates various hardware components of the voice assistance apparatus <NUM>, according to an embodiment of the disclosure.

In an embodiment, the voice assistance apparatus <NUM> includes an interface controller <NUM>, a voice recognizer <NUM>, a voice-based interaction controller <NUM>, a processor <NUM> (e.g., central processing unit (CPU), graphics processing unit (GPU), hardware chipset, etc.) communicatively coupled to a memory <NUM> (e.g., a volatile memory and/or a non-volatile memory). The memory <NUM> includes storage locations configured to be addressable through the processor <NUM>, and a communicator (not shown) configured to communicate internally with aforementioned hardware components, other hardware components, and externally with other components/loT device <NUM> through any of the existing network communication means.

The interface controller <NUM> can be configured to provide an access control of the interface <NUM>. For example, the interface controller <NUM> can include multiple configurations providing an option to select the type of interface <NUM> used to connect/communicate with the other loT device/electronic devices. Further, the interface controller <NUM> can be configured to measure and indicate one or more interface parameters such as, for example, strength of the interface signal from the interface <NUM>, IP address, MAC addresses of the interface <NUM>, and the like.

The voice recognizer <NUM> can be configured to detect voice information received from at least one user associated with each/multiple loT device(s) <NUM>. The voice recognizer <NUM> can also include, for example, a voice sensor and a mike (not shown) configured to receive the voice utterances, a buzzer circuitry/a speaker (not shown) configured to provide an alert indicative signal (audio alert, verbal, etc.,) to the user, an acoustic detector(s), or the like. In an example embodiment, the voice recognizer <NUM> can be a biometric voice sensor configured to authenticate/identify the user based on the voice utterance provided. The constructional feature of the voice recognizer <NUM> is similar to that of any known voice sensor currently existing. For example, the voice recognizer <NUM> can include a membrane portion that detects pressure waves caused by user voice utterance (or, human speech). In an embodiment, the voice recognizer <NUM> can be configured to provide an action in response to detecting the voice utterance.

In another embodiment, the voice recognizer <NUM> can be configured to detect the voice utterance (e.g., at several location (zones) from multiple users) and transmit the voice utterance, detected, to the voice-based interaction controller <NUM>.

The voice-based interaction controller <NUM>, operably coupled with the voice recognizer <NUM>, can be configured to receive the input(s) (e.g., voice utterances) from the voice recognizer <NUM>. In an embodiment, in response to receiving the input(s), the voice-based interaction controller <NUM> can be configured to provide set instructions to the voice recognizer <NUM> to take one or more actions. The one or more actions such as, for example, provide the alert indicative signal, response to the voice utterance, and the like.

Unlike related art methods and systems, the proposed method combines the voice utterances received at multiple loT device <NUM> into a single query, then processes the request.

The memory <NUM> may include one or more computer-readable storage media. The memory <NUM> may include non-volatile storage elements. Examples of such non-volatile storage elements may include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. In addition, the memory <NUM> may, in some examples, be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term "non-transitory" should not be interpreted to mean that the memory <NUM> is non-movable. In some examples, the memory <NUM> can be configured to store larger amounts of information than the memory. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in random access memory (RAM) or cache). The memory <NUM> may be used for storing the user voice utterances captured from loT device(s) <NUM>, storing partially merged utterances along with data such as confidence level etc., completely merged utterances along with data such as confidence level etc., maintain history of user utterances, information regarding loT device(s) <NUM> along with additional data such as their corresponding location etc..

The processor <NUM> controls the general operation of the voice assistance apparatus <NUM>. For example, the processor <NUM> transmits and receives a signal through the interface controller <NUM>. Further, the processor <NUM> records data in the memory <NUM> and reads the recorded data. The processor <NUM> may perform functions of a protocol stack that is required from a communication standard. According to another implementation, the protocol stack may be included in the interface controller <NUM>. To this end, the processor <NUM> may include at least one processor or microprocessor, or may play the part of the processor. Further, the part of the interface controller <NUM> or the processor <NUM> may be referred to as a communication processor (CP).

The processor <NUM> identifies a first voice utterance from a first IoT device in the loT network system, identifies at least one second voice utterance from at least one second IoT device in the loT network system, determines a voice command by combining the first voice utterance and the at least one second voice utterance, and triggers at least one IoT device in the loT network system to perform at least one action corresponding to the voice command. For example, the processor <NUM> may control the voice assistance apparatus <NUM> to perform operations according to the embodiments of the disclosure.

<FIG> shows hardware components of the voice assistance apparatus <NUM> but it is to be understood that other embodiments are not limited thereon. In other embodiments, the voice assistance apparatus <NUM> may include less or more number of hardware components. Further, the labels or names of the hardware components are used only for illustrative purpose and does not limit the scope of the claims. One or more hardware components can be combined together to perform same or substantially similar function in the voice assistance apparatus <NUM>.

<FIG> illustrates various hardware components of the voice-based interaction controller <NUM>, according to an embodiment of the disclosure.

The voice-based interaction controller <NUM> includes an automatic speech recognizer (ASR) <NUM>, a domain classifier (DC) <NUM>, a natural language understanding (NLU) <NUM>, and a merger <NUM>.

The ASR <NUM> can be configured to generate an output indicative of the voice utterances (speech utterances) detected by the voice recognizer <NUM>. The ASR <NUM> can be coupled to a speech interpreter (not shown) configured to manage a digital vocabulary with data items (words, text, keywords, synonyms of keywords, meaning of keywords, or the like) in a memory <NUM>. In an example embodiment, the ASR <NUM> can be configured to communicate with various vocabulary data sources using the network <NUM> for identifying the user(s) voice utterances (first voice utterance, second voice utterance, and the like).

Further, the ASR <NUM> can be configured to intelligently identify/differentiate the various voice utterances provided by the user. For example, voice utterance may vary with respect to the characteristics of each user such as user accent, user emotions, user voice volume, etc. The ASR <NUM> can be configured to convert the derived speech into text using speech-text conversion mechanism (not detailed herein).

The text derived from the ASR <NUM> serves as an input to the DC <NUM>. In an embodiment, the DC <NUM> can be configured to extract the domain associated with speech outputted by the ASR <NUM>. For example, the domain herein may represent one or more subject matter areas, such as messaging, shopping, media, application context based, movie ticket booking, and the like.

Various confidence levels may be defined. The confidence level can be determined based at least one confidence parameter, and the confidence level indicates a level at which a specific parameter can be identified from a voice utterance. The confidence parameters comprise at least one of a domain associated with a voice utterance, an intent associated with a voice utterance, location information associated with a voice utterance, a time at which a voice utterance is received, a user associated with a voice utterance, a context associated with a voice utterance, and a session associated with a voice utterance.

For example, if a first voice utterance received by the first loT device is "send" then a domain confidence level is <NUM>% as the domain is not identified from the first voice utterance, received. Further, if a second voice utterance received by the second loT device is "message" then a domain confidence level is increased from <NUM>% to <NUM>%, as the "message" is classified as one of the domains. Thus, once the domain associated with the multiple partial voice utterance is identified (based on the domain confidence level) then a user intended activity in the classified domain can be computed.

The NLU <NUM> can be configured to perform a match between the one or more partial voice utterances, received, by the voice assistance apparatus <NUM> by computing a probability there between. Thus, based on the probability computed, the NLU <NUM> can be configured to determine the user intend confidence level (to determine the user intended activity) associated with each voice utterance prior to providing the instructions to merge the voice utterances.

For example, if the user of the loT device <NUM> (e.g., 100b) provides the first voice utterance "send a message", then the loT device <NUM> may not be able to understand which message (e.g., SMS, SNS, etc.,) needs to be sent to whom. Thus, the user intend confidence level of the loT device <NUM> with regards to the first voice utterance is minimal (e.g., <NUM>%). Further, if the user again provides the second voice utterance (i.e., through SNS application) to the loT device <NUM> (e.g., 100b), then the user intend confidence level of the loT device <NUM> increases from <NUM>% to <NUM>%. Further, if the user, yet again, provides a third voice utterance i.e., to my friend "Michael", then the user intend confidence level of the loT device <NUM> increases from <NUM>% to <NUM>%. Thus, the NLU <NUM> can be configured to dynamically compute the user intend confidence level for each voice utterance received from multiple loT devices (e.g., loT device 100a, IoT device 100b, IoT device 100c, etc.) based on the probability match computation between the first voice utterance and each successive voice utterance thereof (as shown in <FIG>).

The NLU <NUM> can further be configured to compute the other confidence parameters for the first voice utterance, at least one second voice utterance (includes third voice utterance, fourth voice utterance, and so on), the confidence parameters such as, for example, location information associated with the first voice utterance and at least one second voice utterance received, a time at which the first voice utterance and the at least one second voice utterance is received, a user associated with the first voice utterance and the at least one second voice utterance, a context associated with the first voice utterance and the at least one second voice utterance, and a session associated with the first voice utterance and the at least one second voice utterance.

The output of both the DC <NUM> and the NLU <NUM> can be served as the input to the merger <NUM> to consolidate all the voice utterance in order to determine the voice command. Further, in addition to the output of both the DC <NUM> and the NLU <NUM>, the merger <NUM> also considers the parameters such as, for example, the type of user (authorized/unauthorized), context of the loT device <NUM>, and a context of merger <NUM>.

Once the voice command is processed, the at least one loT device <NUM> in the loT network system <NUM> configured to perform at least one action corresponding to the voice command is selected based on a plurality of parameters. The plurality of parameters such as, for example, at least one of location information of a user, a received signal strength indication (RSSI) information, a head orientation information of the user, information of the loT device <NUM> proximity to the user while the user provides the voice utterance, a voice decibel level associated with at least one of the first voice utterance and the at least one second voice utterance, a noise level associated with the loT device <NUM>, and an availability of the loT device <NUM>.

<FIG> shows hardware components of the voice-based interaction controller <NUM> but it is to be understood that other embodiments are not limited thereon. In other embodiments, the voice-based interaction controller <NUM> may include less or more number of hardware components. Further, the labels or names of the hardware components are used only for illustrative purpose and do not limit the scope of the claims. One or more hardware components can be combined together to perform the same or substantially similar functions in the voice-based interaction controller <NUM>.

<FIG> is a flow diagram illustrating a method for merging the first voice utterance with the at least one second voice utterance, according to an embodiment of the disclosure.

Referring to <FIG>, at operation <NUM>, the method includes converting the voice utterance into text (or, speech) using the ASR technique. In an embodiment, the method allows the ASR <NUM> to convert the voice utterance into text (or, speech) using the ASR technique.

The received voice utterance, and the outputted text by the ASR <NUM>, are then used, at operation <NUM>, as a reference to determine the plurality of parameters based on the received voice utterance and current state of the loT device <NUM>. The plurality of parameters such as i.e., context of IoT device <NUM> (location of the user, mic location, etc.,) speaker recognition (user identification), personnel language modelling (voice decibel (dB) level, voice utterance time, etc.), and sentence structure for a voice utterance.

Further, at operation <NUM>, the method includes determining whether the domain classification success of the text outputted by the ASR <NUM> is possible. In an embodiment, the method allows the DC <NUM> to determine whether the domain classification success of the text outputted by the ASR <NUM> is possible with acceptable confidence levels.

If, at operation <NUM>, the DC <NUM> determines that the domain classification success of the text outputted by the ASR <NUM> is possible with acceptable confidence levels, then at operation <NUM>, the method includes determining whether the NLU can be computed. In an embodiment, the NLU <NUM> can be configured to determine whether the NLU to the outputted text can be computed with acceptable confidence levels.

If, at operation <NUM>, the NLU <NUM> determines that the NLU to the outputted text can be computed with acceptable confidence levels, then, at operation <NUM>, the method includes processing the request for the voice utterance.

If, at operation <NUM>, the DC <NUM> determines that the domain classification success of the text outputted by the ASR <NUM> is not possible with acceptable confidence levels, and at operation <NUM>, the NLU <NUM> determines that the NLU to the outputted text cannot be computed; then the merger <NUM>, at operation <NUM>, can be configured to merge the voice utterance based on the plurality of parameters determined at operation <NUM>.

The possibility of merging voice utterances can be determined based on at least one of the plurality of parameters determined at operation <NUM>. The possibility of merging voice utterances can be construed as whether the voice utterances are successive and/or continuous, or they are distinct from each other. For example, if the voice utterances are successive and/or continuous, they are possibly to be merged, and vice versa. On the other hand, if the voice utterances are distinct, they are impossible to be merged, and vice versa.

The possibility of merging voice utterances can be determined based on whether the user is moving. The processor <NUM> is configured to determine whether the received voice utterance is a voice utterance for a moving user. The processor <NUM> is configured to determine an average received power of each of words composing the received voice utterance. If average received powers for successive words change more than a threshold, the processor <NUM> may determine that the user is moving, and the received voice utterance is a voice utterance for the moving user. The moving state of the user (=a state indicating that the user is moving) can be notified to a wearable device worn by the user, and can be presented to the user through a user interface of the wearable device. If a plurality of voice utterances for the moving users are identified in different loT devices, the plurality of voice utterances can be determined as being possibly to be merged.

The possibility of merging voice utterances can be determined based on a sentence structure of each of received voice utterances. For example, the processor <NUM> can determine whether a sentence identified from each voice utterance is an incomplete sentence a complete sentence. If a plurality of sentences of voice utterances can be complete sentence when merged according to a suitable order even though each of them is incomplete sentence, the voice utterances are possibly to be combined.

The possibility of merging voice utterances can be determined based on a user identification and a voice utterance time. For example, voice information for at least one user can be stored in memory <NUM>. The processor <NUM> can extract voice information from the received voice utterance, compare the extracted voice information with the stored voice information, and determine a user identification for the received voice utterance. Also, the processor <NUM> can identify a voice utterance time for each voice utterance. The voice utterance time may refer to a time at which the voice utterance is received by an IoT device <NUM>. The processor <NUM> can determine whether a plurality of voice utterances with the same user identification are received at an interval less than a threshold interval based on the user identification and the voice utterance time for each voice utterance. If the plurality of voice utterances with the same user identification are received at an interval less than the threshold interval, the plurality of voice utterances can be determined as being possibly to be merged.

The above stated examples of determining merging possibility are just for illustration only. The processor <NUM> can also determine the possibility of merging voice utterances based on at least one of the plurality of parameters determined at operation <NUM> apart from the examples stated above.

The merging of voice utterances may comprise arranging voice utterances that are possibly to be merged in an order of voice utterance time, and concatenating the arranged voice utterances. The merging of voice utterances may further comprise omitting a duplicate part of the voice utterances from at least one of the voice utterances.

The merging of voice utterances may comprise merging the same voice utterances which are received by loT devices at a relatively low power. For example, if the user generating voice utterance is far from all of the loT devices, the voice utterance is received by each loT device at a relatively low power. In this case, a single voice utterance has a low power that is not enough for identifying a voice command or an intend from the voice utterance. Therefore, by merging the same voice utterances which are received by loT devices at a relatively low power, the processor <NUM> can extract a voice command or an intend from the merged voice utterances having a relatively high power.

The merging of voice utterances can be construed as an operation or a series of operations to enhance a confidence level of voice command that can be extracted from at least one of the voice utterances.

Further, at operation <NUM>, the method includes determining whether merging of the voice utterance is possible. If at operation <NUM>, the merger <NUM> determines that the merging of the voice utterance is possible, then, the voice utterance is transmitted to perform the operations disclosed at operations <NUM>-<NUM>. If at operation <NUM>, the merger <NUM> determines that the merging of the voice utterance is not possible, then at operation <NUM>, the method includes providing a notification to the user, notification including i.e., request cannot be processed, wait for next chunk of utterance, or respond failure to user if next utterance is not received with in time.

For example, when the user is moving while talking in the loT environment, then the voice heard at all the loT devices <NUM> will be voice chunks heard when the user is near to that loT devices <NUM>. Hence none of the loT devices <NUM> can independently handle the query of the user. Unlike for related art methods and systems, the proposed method can be used to aggregate all the voice chunks heard at different assistants, and then consolidate all the split utterances captured into a single complete utterance and then find the intent of the user and process the request, and respond back to the user on the best nearby loT device <NUM> while the user ended the last conversation.

The various actions, acts, blocks, steps, or the like in <FIG> may be performed in the order presented, in a different order or simultaneously. Further, in some embodiments, some of the actions, acts, blocks, steps, or the like may be omitted, added, modified, skipped, or the like without departing from the scope of the claims.

<FIG>, <FIG>, and <FIG> are an example of the loT network system in which the voice-based interaction controller processes the multiple voice utterance received by the loT device for providing continuous voice assistance, according to various embodiments of the disclosure.

<FIG> is an example of the loT network system <NUM> in which the voice-based interaction controller <NUM> processes the multiple voice utterance received by the loT device <NUM> (100a, 100b, 100c) for providing continuous voice assistance, according to an embodiment of the disclosure.

Referring to <FIG>, the various steps involved in providing continuous assistance are as detailed herein. The voice-based interaction controller <NUM> can be configured to merge all the portion of voice utterance received at all the loT device <NUM> (100a, 100b, 100c) on time basis (T). Thus, when the user gives the voice utterance comprising a voice command while moving from one location to another location, a portion of the voice command is received by the first IoT device and another portion of the same voice command is received by the second loT device. As seen in <FIG>, when the user gives the voice utterance to the loT device 100a at zone 1A, the voice utterance to the loT device 100b at zone 1B, and the voice utterance to the loT device 100c at zone 1C, then the voice based interaction controller <NUM> (accessible through network <NUM>) can be configured to receive all the voice utterance i.e., partial voice query at time T1, partial voice query at time T2, and partial voice query at time T3 are merged in a time based manner to form a combined voice command. The combined voice command is processed to identify the user intention and response is generated. Referring to <FIG>, the voice-based interaction controller <NUM> can be an Ad-hoc.

<FIG> are example scenarios in which the voice assistance apparatus provides continuous voice assistance using best suitable speaker/loT device <NUM>, according to various embodiments of the disclosure.

Further, if the same voice utterance is heard at multiple loT devices <NUM>, then the voice-based interaction controller <NUM> can be configured to give a preference to best heard loT device <NUM>. Furthermore, the method includes sending the response to the best possible IoT device <NUM> around the user. In an embodiment, the best heard loT device <NUM>/best possible IoT device <NUM> around the user can be selected based on the loT device <NUM> which heard the voice utterance with less noise levels (as shown in <FIG>), the loT device <NUM> in the direction of the receiving voice utterance (as shown in <FIG>), the loT device <NUM> nearest to the user (as shown in <FIG>), the loT device <NUM> based on conflict resolution (as shown in <FIG>), the loT device <NUM> which is not busy (as shown in <FIG>), the loT device <NUM> which heard the voice utterance with maximum voice signal strength, the loT device <NUM> which heard first among all other assistants, etc..

Referring to <FIG>, the voice assistance with wearable device 702a, 702b, and 702c (hereinafter collectively referred to as wearable device <NUM>) is illustrated, according to an embodiment as disclosed herein. The wearable device <NUM> helps in user localization. The wearable device <NUM> aids the user in deciding which assistant (IoT assistant) to be given preference for processing and responding back to the user query (voice utterance). In an embodiment, the user head orientation and the voice signal strength at which the user utters can be determined using the wearable device <NUM> (such as HMD). This orientation helps in directional assistance use cases. If there is no assistant in proximity then the wearable device <NUM> can send voice data over Bluetooth low energy (BLE) to nearby loT device <NUM>. If the loT assistances have been configured to wake up on a particular action such as predefined voice utterance (e.g., "Hey Bixby"), then the wearable device <NUM> provides proximity-based assistant wake up while the user is talking to the loT device(s) <NUM> while moving across zones. When the user is moving across different locations while continuously talking to the loT device <NUM>, then the wearable device <NUM> can send trigger signals to other loT assistants where the user is moving in order to capture the user's next portion utterance without missing any portion of user's utterance and to provide continuity experience.

<FIG> is an example scenario in which the continuous voice assistance in multi-user case is demonstrated, according to an embodiment of the disclosure.

Referring to <FIG>, in this scenario, while placing a single order online, allowing others to continuously add items to the order. The voice assistance apparatus <NUM> consolidates all data, understands intent of the user, and processes the request.

For example, the user at zone-<NUM> provides a first voice utterance <NUM> to the loT device 100a "Place an order for following items on online shopping application. Tomato One Kilo, and Ten Eggs". Further, the user <NUM> moves to the zone-<NUM> and continue to provide to place the order by providing the second voice utterance <NUM> "and an 8GB USB stick" heard by the loT device 100b. Thus, the loT device 100b can be configured to display the order items by merging the first voice utterance and the second voice utterance "Place an order for following items on the online shopping application. Tomato One Kilo, Ten Eggs, and 8GB USB stick". Thus, providing the continuous voice assistance to the user. Similar operations are performed by the loT device 100c at third zone with a third voice utterance <NUM> and any other loT devices at subsequent zones.

For another example, the user at zone-<NUM> provides a first voice utterance <NUM> to the loT device 100a. The intent of the first voice utterance <NUM> may be, for example, writing a text for an e-mail or a text message, or placing an order for item(s) on online shopping. The loT device 100a may identify an identification of the user based on a user identification for the first voice utterance <NUM>. Also, the loT device 100a may detect a device carried by the user (e.g., smart phone, tablet, etc) within a preset distance from the loT device 100a, and may identify an identification of the user based on information on the detected devices. The device carried by the user may comprise a wearable device (e.g., gear device) worn by the user. The loT device 100a transmits at least one of voice data corresponding to the first voice utterance <NUM>, or the identification of the user, to the voice assistance apparatus <NUM>. The loT device 100a can summarize and/or interpret the intent of the first voice utterance <NUM>, and can present information corresponding to the first voice utterance <NUM> though at least one of a display of the loT device 100a, or a speaker of the loT device 100a.

Further, the user may move to the zone-<NUM>, where the loT device 100b exists. The loT device 100b can identify that the user is within a threshold distance from the loT device 100b. For example, the loT device 100b can identify that the user is within a threshold distance from the loT device 100b based on at least one of an indoor positioning system (IPS) information of the device carried by the user, or a signal strength of a signal received from the device carried by the user, or a motion sensor of the loT device 100b. The loT device 100b can report at least one of a time at which the user is identified as being within the threshold distance from the loT device 100b, an identification of the user, or information that someone is in a vicinity of the loT device 100b, to the voice assistance apparatus <NUM>. The voice assistance apparatus <NUM> then may determine whether a first condition or a second condition is satisfied. Herein, the first condition may be defined as that a difference between a time at which information regarding the first voice utterance <NUM> is received by the voice assistance apparatus, and the time at which the user is identified as being within the threshold distance from the loT device 100b, is within a threshold period of time. The second condition may be defined as that the identification of the user identified by the first IoT device 100a is the same as the identification of the user identified by the second loT device. If at least one of the first condition or the second condition is satisfied, the voice assistance apparatus <NUM> may determine that the second voice utterance <NUM> to be provided by the user can be possibly merged with the first voice utterance <NUM>, and can control the loT device 100b to present information corresponding to the first voice utterance <NUM> though at least one of a display of the loT device 100b, or a speaker of the loT device 100b. That is, the user can identify that there is a voice utterance that can be merged, and can identify the intent of the voice utterance. Then, the loT device 100b may receive a second voice utterance <NUM> from the user, and transmit voice data corresponding to the second voice utterance <NUM> to the voice assistance apparatus <NUM>. The voice assistance apparatus <NUM> may merge the first voice utterance <NUM> and the second voice utterance <NUM> based on a possibility of merging as described above.

The voice assistance device <NUM> may control the loT device 100b to present information corresponding to the first voice utterance <NUM> without considering the first condition and the second condition. That is, if the user is just in a vicinity of the loT device 200b, the voice assistance apparatus <NUM> can control the loT device 200b to present information corresponding to the first voice utterance <NUM>.

The loT device 100b may further present a message for inquiring whether to perform a voice merging, in addition to the information corresponding to the first voice utterance <NUM>. The message may be presented through at least one of a display of the loT device 100b, or a speaker of the loT device 100b. The user may determine whether to perform a voice merging with the first voice utterance <NUM>, and may provide a command to the loT device 100b instructing whether to perform a voice merging with the first voice utterance <NUM>. The command may be received by the loT device 100b in a form of a voice recognized by a microphone of the loT device 100b, or a touch input through a display of the loT device 100b. If the loT device 100b received command instructing not to perform a voice merging with the first voice utterance <NUM>, the second voice utterance <NUM> that may be received by the loT device 100b is not merged with the first voice utterance <NUM>. On the other hand, if the loT device 100b received command instructing to perform a voice merging with the first voice utterance <NUM>, the second voice utterance <NUM> that may be received by the loT device 100b is merged with the first voice utterance <NUM>.

All the aforementioned voice utterances may not be limited from a single user but also can be provided/received from multiple and/or different users located at difference zones/locations.

Claim 1:
A method for operating an electronic device managing voice-based interaction in an Internet of things, IoT, network system, the method comprising:
identifying a first voice utterance of a user from a first IoT device among a plurality of IoT devices in the IoT network system;
identifying at least one second voice utterance of the user from at least one second IoT device among the plurality of IoT devices in the IoT network system;
generating a voice command by combining the first voice utterance and the at least one second voice utterance; and
triggering at least one IoT device among the plurality of IoT devices in the IoT network system to perform at least one action corresponding to the voice command,
wherein the at least one second voice utterance is received after the first voice utterance.