ENABLING DEVICE CONTROL PLANNING CAPABILITIES OF SMALL LANGUAGE MODEL

A method for enabling an improved device control capability of a small language model (SLM) transferrable to a hub device configured to be operable by a user in an environment, is disclosed. The method includes performing a fine-tuning the SLM based on a data set including base plans and contrastive plans; generating computer codes corresponding to the fine-tuned SLM; and transferring the generated computer codes to the hub device to be connected with a group of the electronic devices in the environment.

BACKGROUND

The disclosure relates to a system and a method for controlling electronic devices using language models (such as a small language model), for example, home devices or industrial devices.

2. Description of Related Art

Controlling electronic devices (e.g., smart home devices, industrial devices at factories) is a difficult task if the instruction is abstract and the planner needs to adjust dynamic configurations. Language models (in particular, large language model (LLM)) may be used for ‘zero-shot’ planning tasks. That is, based on already containing information, without further training, the LLM may perform high-level tasks such as planning action tasks in accordance with a user's input instruction (e.g., a prompt).

In some cases, (cloud-supported) LLMs may seamlessly perform controlling the electronic devices. However, small language models(SLMs) (e.g., on-device) show limited capabilities for controlling the electronic devices. Examples of the (cloud-supported) LLMs have tens of billions to hundreds of billions of learnable parameters. Examples of SLMs may have only about several billion learnable parameters.

One example of controlling the electronic devices is to control home appliances or devices. For example, a user of the home devices (an automated system connected with a window and a light) provides an ambiguous/abstract/indirect instruction, such as saying “this room is too bright,” then the automated system may (plan to) close the window and/or dim the light.

One example of the automated system is a ‘Smart Home AI Assistant’ configured to perform the above operations on the connected devices (e.g., the window and the light) based on the user instruction. To perform the task of controlling the connected devices, the automated system needs to understand the intent of the user instruction and to develop a plan to achieve the intended goal. It may be complex for the automated system to adjust the plan based on a configuration of an environment (e.g., a house including the window, the light, and/or other available devices) where the automated system is located at.

When working with LLMs, it is important to provide the right ‘prompt’. So-called ‘prompt engineering’ involves carefully designing this input text to guide the LLM towards generating the desired response. ‘In-context learning’ (ICL) takes things a step further than the prompt engineering. The ICL involves adding examples or extra information directly within the prompt. By receiving ‘in-context examples’ from the user, which are used for the ICL of the LLM, the LLM may grasp the task at hand and deliver even more accurate and relevant outputs. There are different flavors of in-context learning; zero-shot, single-shot, and few-shot, which refer to a number of examples provided to the LLM. Recently, the LLMs have shown promising capability as a ‘zero-shot’ action planner in multiple cases. With the aid of proper ‘in-context examples,’ the LLMs may perform aforementioned planning task for different configurations seamlessly.

However, the LLMs may have billions of learnable parameters and may be implemented in servers having powerful computational capabilities, but may not be deployed on (relatively) small devices with limited memory, such as a hub device, a small home appliance, or a smart phone.

One possible solution is to use SLMs that may be deployed on (relatively) small devices as a proxy for LLMs. However, the performance of SLMs may be worse than that of LLMs. SLMs may infer wrong plans to control the electronic devices or may be incapable to plan to control available electronic devices.

SUMMARY

The disclosure is directed to a system and a method for controlling electronic devices (for example, home devices or industrial devices) using language models (such as a large language model (LLM) and a small language model (SLM)).

According to an aspect of the disclosure, a method for enabling an improved device control capability of a small language model (SLM) transferrable to a hub device configured to be operable by a user in an environment, includes: generating, by using a large language model (LLM), a pool of diverse instructions including direct instructions for controlling a first group of electronic devices and indirect instructions for controlling the first group of the electronic devices, wherein the first group of the electronic devices corresponds to all available devices in the environment; generating, by using the LLM, base plans related to operations of controlling the first group of the electronic devices; determining, by using the LLM and a retrieval model, retrieved devices based on the base plans and the indirect instructions, wherein the retrieved devices correspond to a second group of the electronic devices and wherein a first number of the electronic devices in the first group is higher than a second number of the electronic devices in the second group; generating, by using the LLM, contrastive plans based on a third group of the electronic devices, wherein a third number of the electronic devices in the third group corresponds to a number of the first number minus the second number; generating a data set by combining the base plans and the contrastive plans; performing a fine-tuning the SLM based on the data set; generating computer codes corresponding to the fine-tuned SLM; and transferring the generated computer codes to the hub device to be connected with a fourth group of the electronic devices in the environment.

According to an aspect of the disclosure, a method for enabling an improved device control capability of a first language model transferrable to a hub device configured to be operable by a user in an environment, includes: generating, by using a second language model, a pool of diverse instructions including direct instructions for controlling a first group of electronic devices and indirect instructions for controlling the first group of the electronic devices, wherein the first group of the electronic devices corresponds to all available devices in the environment; generating, by using the second language model, first plans related to operations of controlling the first group of the electronic devices; determining, by using the second language model and a retrieval model, retrieved devices based on the first plans and the indirect instructions, wherein the retrieved devices correspond to a second group of the electronic devices and wherein a first number of the electronic devices in the first group is higher than a second number of the electronic devices in the second group; generating, by using the second language model, second plans based on a third group of the electronic devices, wherein a third number of the electronic devices in the third group corresponds to a number of the first number minus the second number; generating a data set by combining the first plans and the second plans; performing a fine-tuning the first language model based on the data set; generating computer codes corresponding to the fine-tuned first language model; and transferring the generated computer codes to the hub device to be connected with a fourth group of the electronic devices in the environment.

According to an aspect of the disclosure, an electronic device for enabling an improved device control capability of a small language model (SLM) transferrable to a hub device configured to be operable by a user in an environment, the electronic device comprising: at least one processor; and at least one memory including computer program code, where the at least one memory and the computer program code are configured, with the at least one processor, to cause the electronic device to at least: generate, by using a large language model (LLM), a pool of diverse instructions including direct instructions for controlling a first group of electronic devices and indirect instructions for controlling the first group of the electronic devices, wherein the first group of the electronic devices corresponds to all available devices in the environment; generate, by using the LLM, base plans related to operations of controlling the first group of the electronic devices; determine, by using the LLM and a retrieval model, retrieved devices based on the base plans and the indirect instructions, wherein the retrieved devices correspond to a second group of the electronic devices and wherein a first number of the electronic devices in the first group is higher than a second number of the electronic devices in the second group; generate, by using the LLM, contrastive plans based on a third group of the electronic devices, wherein a third number of the electronic devices in the third group corresponds to a number of the first number minus the second number; generate a data set by combining the base plans and the contrastive plans; perform a fine-tuning the SLM based on the data set; generate computer codes corresponding to the fine-tuned SLM; and transfer the generated computer codes to the hub device to be connected with a fourth group of the electronic devices in the environment.

DETAILED DESCRIPTION

The terms as used in the disclosure are provided to merely describe specific embodiments, not intended to limit the scope of other embodiments. Singular forms include plural referents unless the context clearly dictates otherwise. The terms and words as used herein, including technical or scientific terms, may have the same meanings as generally understood by those skilled in the art. The terms as generally defined in dictionaries may be interpreted as having the same or similar meanings as or to contextual meanings of the relevant art. Unless otherwise defined, the terms should not be interpreted as ideally or excessively formal meanings. Even though a term is defined in the disclosure, the term should not be interpreted as excluding embodiments of the disclosure under circumstances.

Any of the functions or operations described herein can be processed by one processor or a combination of processors. The one processor or the combination of processors is circuitry performing processing and includes circuitry like an application processor (AP), a communication processor (CP), a graphical processing unit (GPU), a neural processing unit (NPU), a microprocessor unit (MPU), a system on chip (SoC), an IC, or the like.

FIG.1illustrates example components of the electronic device in accordance with an embodiment of the disclosure.

InFIG.1, a (first) electronic device101may communicate with a second electronic device102via a first network198(e.g., a short-range wireless communication network), or a third electronic device104or a server108via a second network199(e.g., a long-range wireless communication network). In one embodiment, the (first) electronic device101may communicate with the third electronic device104via the server108. Throughout the disclosure, the first electronic device101may be referred to as ‘the electronic device101.’ Hereinafter, components of the electronic device101are described. Those components of the electronic device101may be also included in the second electronic device102or the third electronic device104. The first electronic device101, the second electronic device102, or the third electronic device104may be configured to perform methods, steps, or operations described in the disclosure.

In an embodiment, the electronic device101may include a processor120, memory130, an input device150, a sound output circuit155, a display160, an audio circuit170, a sensor176, an interface177, a connection terminal178, a haptic circuit179, a camera180, a power management circuit188, a battery189, a communication circuit190, or an antenna197.

In an embodiment, at least one (e.g., the display160, the sensor176, or the camera180) of the components may be omitted from the electronic device101, or one or more other components may be added in the electronic device101. In an embodiment, some of the components may be implemented as single integrated circuitry. For example, the sensor176(e.g., a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented as embedded in the display160(e.g., a display). In an embodiment, the electronic device101may be a user equipment, a user terminal, a smartphone, a tablet personal computer (PC), a laptop, a PC and/or a server.

In an embodiment, the at least one processor120(or the main processor121or the auxiliary processor123) may be implemented in hardware, firmware, or a combination of hardware and software. The at least one processor120(or the main processor121or the auxiliary processor123) may include one or more of a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a many integrated core (MIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator. The at least one processor120(or the main processor121or the auxiliary processor123) are able to perform control of any one or any combination of the other components of the computing device, and/or perform an operation or data processing relating to communication. The at least one processor120(or the main processor121or the auxiliary processor123) execute one or more programs stored in a memory.

The at least one processor120(or the main processor121or the auxiliary processor123) may be implemented as one or more multi-core processors that include one or more cores (e.g., homogeneous multi-cores or heterogeneous multi-cores). When a plurality of cores are included in the at least one processor120(or the main processor121or the auxiliary processor123), each of the cores includes a cache memory, and a common cache shared by the cores may be included in the at least one processor120(or the main processor121or the auxiliary processor123). Each of the cores may independently read and execute program instructions or each of the cores may read and execute one or more portions of program instructions.

In an embodiment, the at least one processor120(or the main processor121or the auxiliary processor123) may refer to a system-on-a-chip (SoC) in which one or more cores and other electronic components are integrated, a single core processor, a multicore processor, or a core included in the single core processor or the multicore processor, wherein the core may be implemented as a CPU, a GPU, an APU, an MIC, an FPGA, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the disclosure are not limited thereto.

The processor120may execute, for example, software (e.g., a program140) to control at least one other component (e.g., a hardware or software component) of the electronic device101coupled with the processor120, and may perform various data processing or computation. In one embodiment, as at least part of the data processing or computation, the processor120may load a command or data received from another component (e.g., the sensor176or the communication circuit190) in volatile memory132, process the command or the data stored in the volatile memory132, and store resulting data in non-volatile memory134.

In one embodiment, the processor120may include a main processor121(e.g., a central processing unit (CPU) or an application processor (AP)), and an auxiliary processor123(e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor121. Additionally or alternatively, the auxiliary processor123may be adapted to consume less power than the main processor121, or to be specific to a specified function. The processor120may refer to or correspond to one or more processors. For example, the electronic device101may include two or more processors like the processor120. In an embodiment, the main processor121and the auxiliary processor123may comprise processing circuitry.

The auxiliary processor123may be implemented as separate from, or as part of the main processor121. The auxiliary processor123may control at least some of functions or states related to at least one component (e.g., the display160, the sensor176, or the communication circuit190) among the components of the electronic device101, instead of the main processor121while the main processor121is in an inactive (e.g., sleep) state, or together with the main processor121while the main processor121is in an active state (e.g., executing an application). In one embodiment, the auxiliary processor123(e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera180or the communication circuit190) functionally related to the auxiliary processor123.

For example, the processor120of the electronic device101may invoke at least one of the one or more instructions stored in the memory130, and execute the at least one of the one or more instructions, with or without using one or more other components under the control of the processor120. This allows the electronic device101to be operated to perform at least one function according to the at least one instruction invoked. The one or more instructions may include a code generated by a compiler or a code executable by an interpreter. The memory130, which may be a machine-readable storage medium, may be provided in the form of a non-transitory storage medium. Wherein, the term “non-transitory” simply means that the storage medium is a tangible device, and does not include a signal (e.g., an electromagnetic wave), but this term does not differentiate between where data is semi-permanently stored in the memory130(the storage medium) and where the data is temporarily stored in the memory130. In an embodiment, the electronic device101may comprise one or more processors (e.g., the main processor121and the auxiliary processor123), and the one or more instructions may be executed by the one or more processors individually or collectively, thereby causing the electronic device101to perform any combination of one or more operations (or functions, steps) described herein.

In an embodiment, the memory130may include a random-access memory (RAM), a read only memory (ROM), and/or another type of dynamic or static storage device (e.g., a flash memory, a magnetic memory, and/or an optical memory) that stores information and/or instructions for use by the processor120. In an embodiment, the memory130may contain information and/or software related to the operation and use of the electronic device100. For example, the memory130may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optic disk, and/or a solid-state disk), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape, or another type of non-transitory computer-readable medium, along with a corresponding drive.

The memory130may store various data used by at least one component (e.g., the processor120or the sensor176) of the electronic device101. The various data may include, for example, software (e.g., the program140) and input data or output data for a command related thereto. The memory130may include the volatile memory132or the non-volatile memory134. The non-volatile memory134may include the internal memory136or external memory138. The program140may be stored in the memory130as software, and may include, for example, an operating system (OS)142, middleware144, or an application146.

One or more embodiments of the disclosure may be implemented as software (e.g., the operating system142, the application146, the middleware144) including one or more instructions that are stored in the memory130(comprising one or more storage medium) that is readable by the electronic device101.

In an embodiment, the input device150may receive a command or data to be used by another component (e.g., the processor120) of the electronic device101, from the outside (e.g., a user, the second electronic device102, or the third electronic device104) of the electronic device101. The input device150may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

In an embodiment, the sound output circuit155may output sound signals to the outside of the electronic device101. The sound output circuit155may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as playing multimedia or playing recorded data. The receiver may be used for receiving incoming calls. According to some embodiments, the receiver may be implemented as separate from, or as part of the speaker.

In an embodiment, the display160may visually provide information to the outside (e.g., a user) of the electronic device101. The display160may include, for example, a display device, a hologram device, or a projector and control circuitry to control a corresponding one of the display device, hologram device, and projector. According to some embodiments, the display160may include a touch sensor adapted to detect a touch, or a pressure sensor adapted to measure the intensity of force incurred by the touch.

In an embodiment, the audio circuit170may convert a sound into an electrical signal and vice versa. According to an embodiment, the audio circuit170may obtain the sound via the input device150or output the sound via the sound output circuit155or a headphone of an external electronic device (e.g., the second electronic device102or the third electronic device104) directly (e.g., wiredly) or wirelessly coupled with the electronic device101.

In an embodiment, a sensor176may detect an operational state (e.g., power or temperature) of the electronic device101or an environmental state (e.g., a state of a user) external to the electronic device101, and then generate an electrical signal or data value corresponding to the detected state.

In an embodiment, the interface177may support one or more specified protocols to be used for the electronic device101to be coupled with the external entity (e.g., the second electronic device102, the third electronic device104, or the server108) directly (e.g., wiredly) or wirelessly. According to an embodiment, the interface177may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

In an embodiment, the connecting terminal178may include a connector via which the electronic device101may be physically connected with the external electronic device (e.g., the second electronic device102, the third electronic device104, or the server108). According to some embodiments, the connecting terminal178may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

In an embodiment, the camera180may capture a still image or moving images (or a set or one or more still images, or video data). According to some embodiments, the camera180may include one or more lenses, image sensors, ISPs, or flashes.

In an embodiment, the power management circuit188may manage power supplied to the electronic device101. According to some embodiments, the power management circuit188may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

In an embodiment, the battery189may supply power to at least one component of the electronic device101. According to some embodiments, the battery189may include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.

In an embodiment, the communication circuit190may include a transceiver-like component (e.g., a transceiver and/or a separate receiver and transmitter) that enables the electronic device100to communicate with other devices (e.g., the second electronic device102, the third electronic device104, or the server108), such as via a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication circuit190may permit the electronic device100to receive information from another device and/or provide information to another device. For example, the communication circuit190may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, or the like. In an embodiment, the communication circuit190may be a communication ‘interface’ used to connect the electronic device100with the other devices.

In an embodiment, the communication circuit190may include one or more communication processors (CPs) that are operable independently from the processor120(e.g., an application processor) and supports a direct (e.g., wired) communication or a wireless communication. According to an embodiment, the communication circuit190may include a wireless communication circuit192(e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication circuit194(e.g., a local area network (LAN) communication module or a power line communication (PLC) module).

A corresponding one of these communication modules may communicate with the external electronic device via the first network198(e.g., a short-range communication network, such as Bluetooth™, Wi-Fi direct, or IR data association (IrDA)) or the second network199(e.g., a long-range communication network, such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multi components (e.g., multi chips) separate from each other. The wireless communication circuit192may identify and authenticate the electronic device101in a communication network, such as the first network198or the second network199, using subscriber information (e.g., international mobile subscriber identity (IMSI)).

The antenna197may transmit or receive a signal or power to or from the outside (e.g., an external electronic device) of the electronic device101. According to an embodiment, the antenna197may include an antenna including a radiating element composed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna197may include a plurality of antennas (e.g., array antennas).

In an embodiment, a set of components (e.g., one or more components) of the electronic device100may perform one or more functions described as being performed by another set of components of the electronic device100.

One aspect of the disclosure is to enable planning capabilities of controlling multiple devices by an electronic device (such as the first electronic device101, the second electronic device102, or the third electronic device104) (‘device control planning capabilities’) using a small language model (SLM) without using manually annotated device control data. The SLM may generate appropriate plans based on different configurations of places where the electronic devices are located at (e.g., a user's house). One aspect of the disclosure is to use an automated approach to transfer the device control planning capabilities of an LLM to the SLM. The LLM may be used to synthesize ‘instruction-devices-plan’ triplets for device control task automatically in a self-regulatory manner. One aspect of the disclosure is to generate ‘base plans’ and ‘contrastive plans’ by systematically altering the configurations for the same instruction. One aspect of the disclosure is to fine-tune the SLM based on both of the base plans and the contrastive plans, to ensure that the SLM will adjust the device control planning capabilities for different configurations. Throughout the disclosure, the base plans refer to some device controlling capacities corresponding to all available devices, while the contrastive plans refer to other device controlling capacities corresponding to a subset of the all available devices (when at least one of the all available devices becomes unavailable).

FIG.2illustrates an example of an environment for controlling multiple electronic devices. In a house room200shown inFIG.2, a hub device201may be configured to connect with multiple devices, for example, a ceiling light202, a projector203, an air conditioner204, a window206, a ceiling fan208, and a thermostat210. The hub device201may correspond to the first electronic device101, the second electronic device102, or the third electronic device104shown inFIG.1. Non-limiting examples of the hub device201are artificial intelligence (AI) assistant devices, smart phones, and smart home devices. In an embodiment, multiple devices, such as sensors, bulbs, outlets, actuators, or buttons, are placed and connected between a controlling device (the hub device201) and the controlled devices (the ceiling light202, the projector203, the air conditioner204, the window206, the ceiling fan208, and the thermostat210).

The hub device201ofFIG.2may obtain an instruction(e.g. from the server or the user), and then, the hub device201may enable ‘device control planning capabilities’ in accordance with embodiments of the disclosure, and then, the hub device201may control the controlled devices based on the device control planning capabilities.

Throughout this disclosure, the environment of the house room200is described as an example. However, the disclosure and its embodiments are not limited to the house room200. In other words, the embodiments of the disclosure may be implemented or realized in other environments such as industrial places like manufacturing factories.

A set of all available devices (e.g., the ceiling light202, the projector203, the air conditioner204, the window206, the ceiling fan208, and the thermostat210shown inFIG.2) areand a set of all possible instructions from the user are.

In an embodiment, the set of all possible instructions may include direct instructions to directly control at least one specific device, for example, “Turn off the ceiling light.” In an embodiment, the set of all possible instructions may include indirect (abstract) instructions, for example, “this room is too hot.”

For an user instruction u∈and a set of available devices D⊆an aspect of an embodiment is to learn a language model (e.g., SLM)that may come up with ‘n’ necessary device control plans/steps S={si}i=1nto achieve a goal of controlling the devices. Because manually annotating instruction-plan pairs is a tedious task for different configurations in an environment (e.g., home or a factory), a set of annotated {(u, D, S)} triplets (user instruction, device, and plan/step) may be unavailable to fine-tune the language model (e.g., SLM). Thus, as described below, the disclosure proposes an automated system configured to generate the set of annotated {(u, D, S)} triplets (user instruction, device, and plan/step) without manual annotations using the LLM.

FIG.3illustrates diverse instruction generation using an LLM and filter in the loop in accordance with an embodiment of the disclosure.FIG.3Billustrates base plan generation in accordance with an embodiment of the disclosure.

Transfer of knowledge (automatically generated by the LLM) from the LLM to the SLM is described below.

Pretrained SLMs typically do not perform well for device control planning task. However, the LLM may perform well with proper in-context examples. In an embodiment, knowledge from the LLM may be transferred to a SLM to enable better planning capabilities. Towards this goal, the disclosure proposes that the LLM may be used to generate data impression for device control task automatically in a self-regulatory manner. In an embodiment, the LLM may adjust plans for different configurations (e.g., different numbers of devices available at home or in a factory), thus “contrastive plans” may be generated for same instructions with the different configurations. In an embodiment, the generated data may be used to fine-tune the SLM. The overall operations of embodiments in this disclosure are illustrated in the drawings and are described below.

Diverse instruction generation is illustrated inFIG.3Aand described below. A large set of diverse instructions302may be generated using the LLM304. In an embodiment, the process starts with a seed set of manually generated instructions (e.g., 30 samples) (seed instructions306) to generate a large set of diverse instructions302using the LLM304. A pool of generated (diverse) instructions (instruction pool308) may be maintained throughout the operations shown inFIG.3.

Some instructions (e.g., six (6) instructions) may be randomly sampled from the seed instructions306and some instructions (e.g., two (2) instructions) may be randomly sampled from the instruction pool308as “in-context examples” (shown as310inFIG.3B) for the LLM304.

At operation309, the LLM304may generate ‘new’ instructions302based on the randomly sampled instructions received from the seed instructions306and the instruction pool308. At operation312, the generated new instructions302may be forwarded to a filter314configured to perform a filtering on the generated new instructions.

In an embodiment, after the filtering performed by the filter314, one of the generated new instructions302may be added to the instruction pool308. For example, the filter314is a Rogue-L filter. Filtering may correspond to measuring a Rogue-L similarity. In an embodiment, if the Rogue-L similarity of the generated new instruction302with any existing instruction (in the instruction pool308) is less than a threshold (e.g.,0.7), then the generated new instruction302may be moved to and stored at the instruction pool308. If the Rogue-L similarity of the newly generated instruction with any existing instruction is equal to or greater than the threshold (e.g.,0.7), the generated new instruction302may be discarded.

Base plan generation is illustrated inFIG.3Band is described below.

At operations316,318, and320, for each instruction received from the instruction pool308, based on information on all available devices322(i.e., D=), the LLM304may generate a device control plan. At operation316, the ‘in-context examples’310(described above) may be provided to the LLM304. As shown inFIG.3B, the LLM304may generate “base plans”324based on at least three factors, namely, the in-context examples310, the instruction pool308, and the information on all available devices ()322.

In an embodiment, the base plans324may correspond to all available devices ()322. With respect to the embodiment shown inFIG.2, a non-limiting example of the base plans includes: 1. Turn on the ceiling light. 2. Turn on the projector. 3. Turn on the air conditioner. 4. Turn on the window. 5. Turn on the ceiling fan. 6. Turn on the thermostat. 7. Turn off the ceiling light. 8. Turn off the projector. 9. Turn off the air conditioner. 10. Turn off the window. 11. Turn off the ceiling fan. 12. Turn off the thermostat. 13. In an embodiment, above non-limiting examples of the base plans may be generated based on the instruction “The room is too hot”. In an embodiment, the base plans include or correspond to direct instructions and indirect (abstract) instructions.

In an embodiment, the base plans324may be generated considering all the possible (e.g. pre-selected) devices are available.

FIG.2illustrates that one embodiment of the house room200has multiple devices (from the ceiling light202to the thermostat210). In an embodiment of the house room200, it may be possible that some of the multiple devices become unavailable; for example, the ceiling light202and the air conditioner204are broken, thus do not function properly. Thus, an embodiment of the disclosure may propose generating other plans (different from the base plans324) in accordance with different configurations of the environment, such as house room200. Those other plans are the contrastive plans.’

In an embodiment, the blocks (e.g., the seed instructions306to the base plans324) shown inFIG.3may be implemented with computer codes, instructions, or software programs, for example, stored in the memory130, which are executed by the at least one processor120. In an embodiment, the blocks may also be physically implemented by analog and/or digital circuits including one or more of a logic gate, an integrated circuit, a microprocessor, a microcontroller, a memory circuit, a passive electronic component, an active electronic component, and the like, and may also be implemented by or driven by software and/or firmware (configured to perform the functions or operations described herein), which may correspond to the components illustrated inFIG.1. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like. Circuits included in a block may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware to perform some functions of the block and a processor to perform other functions of the block. Each block of the embodiments may be physically separated into two or more interacting and discrete blocks. Likewise, the blocks of the embodiments shown inFIG.3may be physically combined into more complex blocks.

Relevance device retrieval and contrastive plan generation are illustrated inFIG.4and are described below.

For example, in a scenario, when the user instruction is “the room is too hot,” a plan (based on the base plans324) may be an operation of turning on the air conditioner204. In the scenario, however, the air conditioner204may be unavailable to use, instead the ceiling fan208is available. In an embodiment, it may be desirable to update the plan to include an operation of turning on the ceiling fan208.

One possible way is to randomly consider different sets of the available devices and use the LLM304to generate additional plans. However, there may be too many possible combinations of all available devices. To learn the configuration dependency, abundant examples may need to be sampled from different combinations of the all available devices. Instead, an embodiment of the disclosure consider different available device combinations based on the base plans324.

First, in an embodiment, the LLM304may be used to divide all the instructions into two types: i) direct instructions, ii) indirect (abstract) instructions. The direct instructions may refer to one or more controlled devices, while the indirect (abstract) instructions do not refer to any specific devices. An example of the direct instructions is “turning on air conditioner,” while an example of the indirect (abstract) instructions is “this room is too hot.”

InFIG.4A, operation402indicates that the instruction404is input to the LLM304. In an embodiment, at operation406, the LLM304may determine whether the instruction404is an indirect (abstract) instruction. When the instruction404is determined (by the LLM304) that the instruction404is indeed the indirect (abstract) instruction408(operation410), then a retrieval model412may be used to generate a set of ‘retrieved devices,’ based on the indirect (abstract) instruction408and the base plans324(received at operation409). The retrieval model may be functionally connected with the LLM304. An example of the retrieval model may be pretrained sentence transformer.

In an embodiment, different plans may be generated based on user preferences and the availability of devices. As shown inFIG.4A, only indirect (abstract) instructions408may be considered for generating contrastive plan generation. The embodiments of the disclosure are to remove some devices from the set of all possible devices322, based on the base plans324. When the instruction404is the indirect (abstract) instruction408, based on the base plans324, the retrieval model412is used to generate the set of the retrieve devices414corresponding to the indirect (abstract) instruction408. In an embodiment, the retrieval model4012may identify the set of the retrieve devices414being used in the base plans324.

As shown inFIG.4B, at operations416,418, and420, the LLM304may be used to generate the contrastive plans422, based on the indirect (abstract) instruction408, the in-context examples310, and a reduced set of available devices424(the all available devices322minus (−) the retrieved devices414). In an embodiment, the LLM304may be used to generate new plans (namely, the contrastive plans422), which are different from the base plans324, for the same indirect (abstract) instruction408and the reduced set of available devices424. In an embodiment, the reduced set of available devices424may refer to an updated list of available devices. By using the reduced set of available devices424, the LLM203may generate a plan for the same instruction with a different set of available devices. In an embodiment, contrastive plans422may adjust planning for different home configurations. In an embodiment, contrastive plans422may lead to better sensitivity of the LLM304for different home configurations.

In an embodiment, the blocks (e.g.,406to422) shown inFIG.4AandFIG.4Bmay be implemented with computer codes, instructions, or software programs, for example, stored in the memory130, which are executed by the at least one processor120. In an embodiment, the blocks shown inFIG.4may also be physically implemented by analog and/or digital circuits including one or more of a logic gate, an integrated circuit, a microprocessor, a microcontroller, a memory circuit, a passive electronic component, an active electronic component, and the like, and may also be implemented by or driven by software and/or firmware (configured to perform the functions or operations described herein), which may correspond to the components illustrated inFIG.1. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards and the like. Circuits included in a block may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuitry), or by a combination of dedicated hardware to perform some functions of the block and a processor to perform other functions of the block. Each block of the embodiments may be physically separated into two or more interacting and discrete blocks. Likewise, the blocks of the embodiments shown inFIG.4may be physically combined into more complex blocks.

Fine-Tuning of a SLM is Described Below.

As discussed above, the LLM304may be used to generate both of the base plans324and the contrastive plans422for a diverse set of instructions (including the direct instructions and/or the indirect (abstract) instructions). A first set of instruction-device-plan triplets with the base plans324and a second set of instruction-device-plan triplets with the contrastive plans422may be combined to prepare a device control data set. For fine-tuning of the SLM502, the device control data set may be randomly split into three different sub-sets, namely, (i) a training set, (ii) a validation set, and (iii) a testing set.

In an embodiment, N triplets {(ui, Di, Si)}i=1Nmay be prepared for fine-tuning the SLM502with weights Θ. A standard autoregressive language modeling, which is known in related art, may be used to ‘fine-tune’ the SLM502, which aims to maximize the following likelihood:=Σi=1N[log P(Si|ui, Di; Θ)]. Here, uiis a user instruction, Diis a set (a group) of available devices, and Siis a device control plan/step. As known in the related art, the conditional probability P may be modeled using a neural network with parameters (weights Θ).

The above-described operations for the fine-tuning of the SLM502may not be performed by human mental process or human manual operations with pens and paper. Rather, those operations may need to be performed by a specialized (not generic) computer that may have components of the electronic device101, for example, the processor120and the memory130.

FIG.5illustrates embodiments about practical applications of the disclosure. A vendor (a manufacturer, a programmer, a supplier) of computer programs or computer codes may perform operations (504to516) shown inFIG.5, by using the electronic device101ofFIG.1. In an embodiment, the operations may be performed by the processor120, the memory130, and/or other components of the electronic device101ofFIG.1, which is operated by the vendor.

In an embodiment, operation504may correspond to the diverse instruction generation shown inFIG.3Aand described above. In an embodiment, operation506may correspond to the base band generation shown inFIG.3Band described above. An output of operation506is the base plans324.

In an embodiment, operation508may correspond to the relevance device retrieval shown inFIG.4Aand described above. In an embodiment, operation510may correspond to the contrastive plan generation shown inFIG.4Band described above. An output of operation510is the contrastive plans422.

In an embodiment, at operation512, the base plans324and the contrastive plans422may be combined to generate a data set. In an embodiment, the data set may be a set of instructions-devices-plans triplets. At operation514, the SLM502may be fine-tuned with the data set. At operation516, computer codes corresponding to the fine-tined SLM502may be generated.

At operation518, the generated computer codes may be transferred to the hub device201. In an embodiment, the vendor may install the generated computer codes into the hub device201that is sold to the user. In an embodiment, the user may download the generated computer codes from a cloud system or the server108, where the generated computer codes are stored by the vendor.

The hub device201may correspond to the first electronic device101. In an embodiment, the hub device201may have the input device150. After the generated computer codes are installed or downloaded in the hub device201, the hub device201may receive, via the input device150, a verbal instruction (utterance such as ‘this room is too hot’ or ‘turn off the air conditioner’) from a user of the hub device201. The generated computer codes may be stored in the memory130and may be executed by the processor120. Based on the received verbal instruction, the generated computer codes (corresponding to the fine-tuned SLM502) may provide an output (or a response) to the received verbal instruction. Then, the output (the response) may be forwarded to application program interfaces (APIs) of the controlled devices (e.g., inFIG.2andFIG.5,201to210), which are stored in the memory130and/or in memories of the controlled devices.

FIG.6illustrates operations for generating computer codes corresponding to a fine-tuned SLM having improved device control capabilities and transferring the generated computer codes to a hub device, in accordance with some embodiments of the disclosure.

At operation600, the operations include generating, by using an LLM, a pool of diverse instructions including direct instructions for controlling a first group of electronic devices and indirect instructions for controlling the first group of the electronic devices. The first group of the electronic devices corresponds to all available devices in the environment.

At operation602, the operations include generating, by using the LLM, base plans related to operations of controlling the first group of the electronic devices.

At operation604, the operations include determining, by using the LLM and a retrieval model, retrieved devices based on the base plans. The retrieved devices correspond to a second group of the electronic devices and a first number of the electronic devices in the first group is higher than a second number of the electronic devices in the second group.

At operation606, the operations include generating, by using the LLM, contrastive plans based on a third group of the electronic devices. A third number of the electronic devices in the third group corresponds to a number of the first number minus the second number

At operation608, the operations include generating a data set by combining the base plans and the contrastive plans.

At operation610, the operations include performing a fine-tuning the SLM based on the data set.

At operation612, the operations include generating computer codes corresponding to the fine-tuned SLM.

At operation614, the operations include transferring the generated computer codes to the hub device to be connected with a fourth group of the electronic devices in the environment. In an embodiment, descriptions of the data flow diagram described above with reference toFIG.6may be omitted for the sake of brevity. In an embodiment, descriptions of the data flow diagram described above with reference toFIG.6may be used to implement at least a portion of at least one of the example application of first electronic device101and may include additional feature.

Improvements to related computer technologies about controlling devices by language models are described below.

A SLM of the related art may be unable to do device control planning with in-context examples. Whenever prompted with instruction and available list of devices, the SLM may generate steps mostly involving all the available devices, even when the devices are not relevant to the instruction. An LLM of the related art may not generate irrelevant steps for the prompted instruction. However, the LLM may be too resource-demanding, thus may not be transferred to any hub device operatively connected with multiple devices controlled by the hub device. In contrast, the SLM may be relatively lighter, and thus, may be transferrable to the hub device and may work as an on-device AI assistant. In summary, the LLM may have device controlling capabilities, but the LLM may not be transferred to or be installed in an electronic device such as a hub device controlling multiple controlled devices in an environment. The SLM may be transferred to or be installed in the electronic device. However, the SLM may not have device controlling capabilities that properly control the multiple controlled devices in the environment.

The above-described embodiments of the disclosure provide particular technical solutions to the above-noted problems of the related language models, the LLM and SLM. In an embodiment of the disclosure, the SLM is fine-tuned with synthesized triplets having both of the base plans and the contrastive plans, as discussed above. That is, the SLM, which is fine-tuned in accordance with an embodiment of the disclosure, may have device controlling capabilities that properly control the multiple controlled devices in the environment. Also, in an embodiment of the disclosure, the SLM may be transferred to or be installed in the electronic device such as the hub device controlling multiple controlled devices in the environment.

In an embodiment, the above-described embodiments of the disclosure may generate SLM for device control with only abstract instruction. In an embodiment, the SLM may be on-device. In an embodiment, the SLM may be generated by fine-tuning the SLM model with the obtained data set with 1) diverse instruction generation, 2) base plan generation and 3) contrastive plan generation.

In an embodiment, the above-described embodiments of the disclosure may be used in a single room or/and a house with multiple rooms. In an embodiment, the above-described embodiments of the disclosure may be used to perform planning tasks where plans are inter-dependent on devices from different rooms.

In an embodiment, the above-described embodiments of the disclosure may accommodate the need of user preferences. Preferences may vary from user to user. The disclosure may be equipped with online adaptation system that will learn from user interactions and preferences. In an embodiment, the generated plan may be more personalized.

For example, computer codes corresponding to the SLM fine-tuned with the synthesized triplets (having both of the base plans and the contrastive plans) result in significant improvement of evaluation scores (obtained by well-known evaluation metrics such as BLEU and ROUGE) than other computer codes corresponding to the SLM fine-tuned with data set having only the base plans or other computer codes corresponding to the SLM fine-tuned with data set having the base plans and random plans.

According to one or more embodiments, in a non-volatile storage medium storing instructions, the instructions may be configured to, when executed by at least one processor, cause the at least one processor to perform at least one operation. The at least one operation may include displaying an application screen of a running application on a display, identifying a data input field included in the application screen, identifying a data type corresponding to the data input field, displaying at least one external electronic device, around the electronic device, capable of providing data corresponding to the identified data type, receiving data corresponding to the identified data type from an external electronic device selected from among the at least one external electronic device through a communication circuit, and entering the received data into the data input field.

The embodiments of the disclosure described in the present specification and the drawings are only presented as specific examples to easily explain the technical content according to the embodiments of the disclosure and help understanding of the embodiments of the disclosure, not intended to limit the scope of the embodiments of the disclosure. Therefore, the scope of one or more embodiments of the disclosure should be construed as encompassing all changes or modifications derived from the technical spirit of one or more embodiments of the disclosure in addition to the embodiments disclosed herein.