EDGE COMPUTING SYSTEM AND METHOD

An electronic device includes: a communication module; a memory; and a processor configured to: receive a connection test command for a leaf device positioned in a network from an external server by using the communication module; based on receiving the connection test command, identify whether a connection to the leaf device is possible, by using the communication module, and transmit a result of the identification to the external server; based on identifying that the connection to the leaf device is possible, receive at least one module configured to perform an edge computing service from the external server, and store and install the at least one module in the memory; and execute the installed at least one module to perform the edge computing service, based on data received from the leaf device.

BACKGROUND

The disclosure relates to an edge computing system and, more specifically, an edge computing system including a leaf device, an edge device, and a cloud server, and a method for performing an edge computing service.

2. Description of Related Art

Cloud computing technology is a technology providing a computing resource existing at a location different from that of a user, to the user over a network to provide a computing service such as a server, a storage, software, or analysis. Data or contents of a user collected in an Internet of Things (IoT) device is stored in a cloud server, and a cloud service is provided to a user through processing of data.

In order to process data generated in an IoT system, cloud computing is used. In cloud computing, according to an increase of the number of devices configured for an IoT system and the amount of data, a load occurs in a system, and a security or privacy issue may occur in a data transfer and/or storage process. Furthermore, if an error occurs in a cloud server or there is no Internet connection, service provision is not possible.

In order to solve these problems of cloud computing, edge computing technology is used. Edge computing is an open architecture for extending a part of cloud computing and service to a device existing at an edge of a network. For example, when edge computing technology is used in an IoT system, an application or a service for which low latency and privacy are important among applications or services processed in a cloud server may be partially dispersedly processed on an edge device.

Edge computing technology is used in IoT systems. However, conventional edge computing service is provided via a fixed module to a fixed edge device. For example, an edge device connected to at least one leaf device and including software and/or hardware for processing data is required to exist in a home network.

According to the related art technology, an edge computing service might not be available according to a state (e.g., movement or lack of resource) of an edge device, and resources for edge computing are required to be secured.

SUMMARY

Provided is an edge computing system and an edge computing method by which edge computing is implementable using a device having idle computing power in a network.

Further, provided is an edge computing system and an edge computing method by which a module and/or service capable of edge computing is installed according to an electronic device registered in an IoT service (e.g., SmartThings™) and/or devices that a user has.

Further still, provided is an edge computing system and an edge computing method using devices that a user has, as a leaf device or an edge device without a user input for use as a designated device (e.g., a leaf device or an edge device).

According to an aspect of the disclosure, an electronic device includes: a communication module; a memory; and a processor configured to: receive a connection test command for a leaf device positioned in a network from an external server by using the communication module; based on receiving the connection test command, identify whether a connection to the leaf device is possible, by using the communication module, and transmit a result of the identification to the external server; based on identifying that the connection to the leaf device is possible, receive at least one module configured to perform an edge computing service from the external server, and store and install the at least one module in the memory; and execute the installed at least one module to perform the edge computing service, based on data received from the leaf device.

The processor may be further configured to transmit device information including at least one of identification information, position information, and network information of the electronic device to the external server to request registration of the electronic device.

The processor may be further configured to: identify at least one external device positioned in a network identical to that of the electronic device by using the communication module; transmit identification information of the at least one external device identified using the communication module to the external server; and receive a connection test command for the leaf device selected by a user device or the external server among the at least one external device.

The at least one module includes a device module configured to receive data transmitted from the leaf device and transmit the received data to the external server.

The processor may be further configured to, based on receiving data from the leaf device according to a first protocol, transmit the data to the external server according to a second protocol by using the device module.

The at least one module may include a service module configured to perform the edge computing service, based on the received data.

The service module may be configured to analyze video data received from the leaf device to perform a service including at least one of object recognition and object tracking.

The connection test command may be received from an Internet of Things (IoT) management server, and the at least one module may be received from an IoT hub server.

According to an aspect of the disclosure, an edge computing service support method of an Internet of Things (IoT) server, includes: transmitting, to at least one of an edge device and a leaf device selected by a user device, a test command for identifying whether the edge device and the leaf device are connectable to each other; based on identifying that that the edge device and the leaf device are connectable, determine at least one module required for performing an edge computing service, based on device information of the edge device and the leaf device; and transmitting the at least one module to the edge device, or requesting an IoT hub server to transmit the at least one module to the edge device.

The edge computing service support method may further include: receiving device information of the edge device from one of the user device and the edge device, and registering the edge device; and receiving device information of the leaf device from one of the user device and the leaf device and registering the leaf device.

The determining the at least one module required for performing the edge computing service may include: identifying a performable edge computing service to correspond to device information of the edge device and device information of the leaf device; and determining at least one module required for performing the edge computing service on a database.

The at least one module may include a device module configured to enable the edge device to receive data transmitted from the leaf device and transmit the received data to an external server, and a service module configured to enable the edge device to analyze data transmitted from the leaf device to perform the service.

The service module may be configured to analyze video data received from the leaf device to perform a service including at least one of object recognition and object tracking.

The edge computing service support method may include: based on the edge device not being connected to the leaf device, transmitting or receiving control information through a first channel connected to the leaf device, and receiving data obtained by the leaf device through a second channel connected to the leaf device; and based on the edge device being connected to the leaf device, maintaining the first channel and receiving data obtained by the leaf device from the edge device.

According to an aspect of the disclosure, an edge computing method of an electronic device, includes: receiving a connection test command for a leaf device positioned in a network from an external server; based on the receiving the connection test command, identifying whether a connection to the leaf device is possible, and transmitting a result of the identifying to the external server; based on identifying that the connection to the leaf device is possible, receiving at least one module configured to perform an edge computing service from the external server, and storing and installing the at least one module in a memory; and executing the installed at least one module to perform the edge computing service, based on data received from the leaf device.

The edge computing method may further include transmitting device information including at least one of identification information, position information, and network information of the electronic device to the external server to request registration of the electronic device.

The at least one module may include a device module configured to receive data transmitted from the leaf device and transmit the received data to the external server.

The executing the installed at least one module to perform the edge computing service may include transmitting data received from the leaf device according to a first protocol to the external server according to a second protocol by using the device module.

The at least one module may include a service module configured to perform the edge computing service, based on the received data.

According to various embodiments of the disclosure, edge computing may be implemented using a device having idle computing power in a network. Various other advantageous effects identified explicitly or implicitly through the disclosure may be provided.

DETAILED DESCRIPTION

Hereinafter, various embodiments disclosed in the present disclosure will be described with reference to the accompanying drawings. For convenience of explanation, the size of the components shown in the drawings may be exaggerated or reduced, and embodiments consistent with the disclosure are not necessarily limited by the drawings.

FIG.1illustrates devices in an example IoT environment (or IoT system) according to various embodiments.

Referring toFIG.1, an IoT system (or edge computing system)100may include at least one leaf device120, at least one edge device110, a user device130, and a cloud network140. For example, the leaf device120, the edge device110, and the user device130may be arranged at adjacent positions (e.g., in home) and connected to the same home network (e.g., the same access point (AP)), and the cloud network140may be remotely located and connected to the leaf device120, the edge device110, and the user device130through Internet.

In the disclosure, according to a function or an operation of each device in the IoT system100, devices may be classified as the leaf device120, the edge device110, and the user device130, and then described. However, the same device (e.g., a smartphone or a tablet PC) may operate as one of the leaf device120, the edge device110, and the user device130in some cases. In other words, a name or a definition of a device described in the disclosure does not limit a function and/or an operation of the device.

According to various embodiments, the leaf device120is an end point of the IoT system100, and may use a sensor to collect various data and transmit same to the edge device110or the cloud network140. In addition, the leaf device120may perform various operations according to a command transferred from the cloud network140or the user device130. Referring toFIG.1, a device, such as a camera121, a refrigerator122, a bulb123a, or a digital thermometer123b, may be the leaf device120.

According to various embodiments, the leaf device120may access the cloud network140through Internet, a device (e.g., the bulb123aor the digital thermometer123b) not supporting an Internet protocol (IP) among the leaf devices120may transmit sensed data to a hub device124via supported non-IP-based communication (e.g., Bluetooth™ or Zigbee™), and the hub device124may transmit sensing data of each leaf device123aand123bto the cloud network140through Internet. A detailed configuration and operation of the leaf device120will be described in more detail with reference toFIG.5.

According to various embodiments, the cloud network140may include various server devices (e.g., an IoT management server and an IoT hub server) located on the network to support a cloud computing service in the IoT system100. The cloud network140may perform computing processing of sensing data received from the leaf device120, and transmit a command for controlling the leaf device120.

According to various embodiments, the cloud network140may perform a function of operating and managing a particular device in a home network to operate as the edge device110. For example, the cloud network140may include an IoT server (e.g., an IoT management server or an IoT hub server), and the IoT server may perform an edge computing service, such as registration, connection, or management of the edge device110and the leaf device120, and provide a module (e.g., a device module or a service module) required for the edge computing service to the edge device110.

According to various embodiments, the edge device110may process, by itself, data received from the leaf device120, or transmit same to the cloud network140(e.g., IoT server). The edge device110may be a device, such as a TV112or a tablet PC111, including a hardware and/or software resource required for an edge computing service. The edge device110may be connected to the cloud network140through Internet, and may establish a home network with the leaf device120.

According to various embodiments, multiple edge devices110may exist in the home network, and the leaf device120may be connected to one of the multiple edge devices110and transmit data thereto. For example, when a particular leaf device120is connected to the edge device110, the edge device may download a module (e.g., a device module or a service module) required for an edge computing service from the cloud network140and execute the same module.

According to various embodiments, the edge device110may perform a device unique function (e.g., a video output function of a TV), and may perform an edge computing service through a hardware and/or software resource at least partially simultaneously with performing the unique function, or for an idle time for which the unique function is not performed.

A detailed configuration and operation of the edge device110will be described in more detail with reference toFIG.4.

According to various embodiments, the user device130may provide various user interfaces related to an edge computing service through an application. For example, the user device130may display, on a display, a data (e.g., camera video streaming) obtained by the leaf device120, or result data (e.g., person recognition) obtained by processing the data by the edge device110or the cloud network140. In addition, the user device130may receive a user input, such as connection of the edge device110and/or the leaf device120, or registration thereof in a server, and transmit the same user input to the cloud network140. A detailed configuration and operation of the user device130will be described in more detail with reference toFIG.3.

In various embodiments of the disclosure, an edge computing service may be dynamically implemented using a device having idle computing power in a home network. To this end, when a device is determined as the edge device110for a particular leaf device120in a home network, the IoT system100may download and install, in the edge device110, a module (e.g., a device module or a service module) required for an edge computing service, and perform the edge computing service by using corresponding modules. Accordingly, unlike a conventional technology providing an edge computing service to a fixed device via a fixed module, the edge device110may be dynamically configured.

FIG.2AandFIG.2Billustrate devices of an example edge computing system according to various embodiments.

Referring toFIG.2A, an edge computing system may include a leaf device220, an edge device210, a user device230, an IoT hub server250, and an IoT management server240. As described with reference toFIG.1, various IoT devices may exist on a home network, and inFIG.2, one leaf device (e.g., the camera121inFIG.1) and one edge device (e.g., the TV112inFIG.1) are described as an example.

According to various embodiments, the IoT management server240(e.g., a SmartThings™ server) is a server device that provides various services for determination, connection, and/or operation of an edge computing service, and may include a provision manager242, a module manager244, and an edge-leaf manager246.

According to various embodiments, the provision manager242may perform a function of relaying in the middle of the edge device210and the IoT hub server250to be connected to each other. For example, when the edge device210is initially registered in the IoT management server240, the provision manager242may transmit, to the edge device210, a connection string allowing the edge device210to be connected to the IoT hub server250.

According to various embodiments, the module manager244may manage pieces of information on various modules provided for an edge computing service and a device supporting each service. The module required for performing an edge computing service may include a device module219that allows the edge device210to transmit data transmitted from the leaf device220, to an external server (e.g., the IoT hub server250), and a service module218including programs executed to implement a service in the edge device210, based on data transmitted from the leaf device220.

According to various embodiments, the edge-leaf manager246may manage a connection state between the edge device210and the leaf device220existing in several home networks. For example, if the leaf device220and the edge device210registered in the IoT management server240are connected to each other or disconnected from each other, the edge device210and/or the leaf device220may transmit connection or disconnection information to the IoT management server240, and the IoT management server240may store information about which edge device210and which leaf device220are connected to each other in real time, and which service is being performed.

According to various embodiments, the IoT hub server250may support a cloud computing platform, and provide data required for connection between the leaf device220and the edge device210existing in a cloud environment. The IoT hub server250may include an IoT hub252and a module registry254.

According to various embodiments, the module registry254may be a storage of modules (e.g., the device module219and the service module218) required for performing an edge computing service.

According to various embodiments, the IoT hub252may maintain a connection with the edge device210, provide a module stored in the module registry254to the edge device210, and maintain information of modules installed in several edge devices210.

According to various embodiments, the edge device210(e.g., the edge device110inFIG.1) may be a device, such as a TV, a tablet PC, or a laptop PC, having a device unique function and including a hardware and/or software configuration (e.g., an edge runtime or a basic module) for an edge computing service. The edge device may perform an edge computing service through a hardware and/or software resource at least partially simultaneously with performing the unique function, or for an idle time for which the unique function is not performed.

According to various embodiments, the edge device210may include an interface212for communication with a cloud (e.g., the IoT management server240or the IoT hub server250), an operating system (OS)214, and an edge runtime216. For example, the edge device210may need a hardware condition (e.g., CPU performance) for operating the operating system214, and may be configured as a real time operating system (RTOS).

The edge runtime216and a basic module for edge computing may be installed in the edge device210through a software upgrade or a processing process of the edge device210. The edge runtime216may include a daemon program for interworking with an IoT server, and the basic module is a program required for communication with the IoT server and may be configured as a container. For example, the basic module may be a container installed in an environment of the edge runtime216.

According to various embodiments, when the edge device210is connected to a particular leaf device220, the edge device210may receive and install at least one module for performing an edge computing service from the IoT hub server250.

For example, the at least one module may be determined according to the type of the connected leaf device220and/or the type of a performable service, and may include the device module219corresponding to the corresponding leaf device220and/or the service module218corresponding to the type of a service to be performed. If the edge device210is connected to multiple leaf devices220, the device module219(e.g., first device module219aand second device module219b) corresponding to each leaf device220may be installed. The edge device210may execute the edge runtime216in a provision process to be connected to the IoT hub server250, and the at least one module may be additionally installed and executed according to the type of the leaf device220. The edge device210may activate or deactivate an edge mode according to a command received from the IoT hub server250or the IoT management server240. If an edge mode is deactivated, the edge device210may perform only a unique function (e.g., a video output function of a TV), and the device module219and the service module218need not be executed.

According to various embodiments, the leaf device220(e.g., the leaf device120inFIG.1) may transmit data obtained using a sensor to the connected edge device210or a cloud network (e.g., the IoT management server240or the IoT hub server250). For example, an Internet protocol (IP) camera operating as the leaf device220may be connected to the edge device210to transmit video streaming the edge device210.

According to various embodiments, the user device230may be a device, such as a smartphone or a tablet PC, which is capable of executing various applications and includes a display capable of displaying a user interface (UI). The user device230may install and/or execute an application for an edge computing service, and receive contents and a notification generated in the leaf device220through the corresponding application. When the edge device210and the leaf device220are connected to each other, contents and a notification generated in the leaf device220may be transmitted to the user device230via the edge device210.

According to various embodiments, a function of the IoT hub server250and the IoT management server240may be performed by one server device (e.g., the IoT server260inFIG.2B). For example, referring toFIG.2B, the IoT server260may include, as elements of the IoT hub server250and the IoT management server240described above, an IoT hub261(e.g., the IoT hub252inFIG.2A), a module registry262(e.g., the module registry254inFIG.2A), a provision manager263(e.g., the provision manager242inFIG.2A), a module manager264(e.g., the module manager244inFIG.2A), and an edge-leaf manager265(e.g., the edge-leaf manager246inFIG.2A).

Alternatively, the functions may be performed by three or more multiple server devices. For example, each element of the IoT hub server250and the IoT management server240inFIG.2Amay be dispersedly arranged by three or more multiple server devices existing on a network, or some operations performed by each element may be dispersedly performed by several sever devices.

FIG.3is a block diagram illustrating a user device in a network environment according to various embodiments. Hereinafter, a user device (e.g., the user device130inFIG.1or the user device230inFIG.2A) of an edge computing system may also be referred to as an electronic device301.

Referring toFIG.3, the electronic device301in a network environment300may communicate with an electronic device302via a first network398(e.g., a short-range wireless communication network), or at least one of an electronic device304or a server308via a second network399(e.g., a long-range wireless communication network). According to an embodiment, the electronic device301may communicate with the electronic device304via the server308. According to an embodiment, the electronic device301may include a processor320, memory330, an input module350, a sound output module355, a display module360, an audio module370, a sensor module376, an interface377, a connecting terminal378, a haptic module379, a camera module380, a power management module388, a battery389, a communication module390, a subscriber identification module (SIM)396, or an antenna module397. In some embodiments, at least one of the components (e.g., the connecting terminal378) may be omitted from the electronic device301, or one or more other components may be added in the electronic device301. In some embodiments, some of the components (e.g., the sensor module376, the camera module380, or the antenna module397) may be implemented as a single component (e.g., the display module360).

The processor320may execute, for example, software (e.g., a program340) to control at least one other component (e.g., a hardware or software component) of the electronic device301coupled with the processor320, and may perform various data processing or computation. According to one embodiment, as at least part of the data processing or computation, the processor320may store a command or data received from another component (e.g., the sensor module376or the communication module390) in volatile memory332, process the command or the data stored in the volatile memory332, and store resulting data in non-volatile memory334. According to an embodiment, the processor320may include a main processor321(e.g., a central processing unit (CPU) or an application processor (AP)), or an auxiliary processor323(e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor321. For example, when the electronic device301includes the main processor321and the auxiliary processor323, the auxiliary processor323may be adapted to consume less power than the main processor321, or to be specific to a specified function. The auxiliary processor323may be implemented as separate from, or as part of the main processor321.

The auxiliary processor323may control at least some of functions or states related to at least one component (e.g., the display module360, the sensor module376, or the communication module390) among the components of the electronic device301, instead of the main processor321while the main processor321is in an inactive (e.g., sleep) state, or together with the main processor321while the main processor321is in an active state (e.g., executing an application). According to an embodiment, the auxiliary processor323(e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module380or the communication module390) functionally related to the auxiliary processor323. According to an embodiment, the auxiliary processor323(e.g., the neural processing unit) may include a hardware structure specified for artificial intelligence model processing. An artificial intelligence model may be generated by machine learning. Such learning may be performed, e.g., by the electronic device301where the artificial intelligence is performed or via a separate server (e.g., the server308). Learning algorithms may include, but are not limited to, e.g., supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-network or a combination of two or more thereof but is not limited thereto. The artificial intelligence model may, additionally or alternatively, include a software structure other than the hardware structure.

The memory330may store various data used by at least one component (e.g., the processor320or the sensor module376) of the electronic device301. The various data may include, for example, software (e.g., the program340) and input data or output data for a command related thererto. The memory330may include the volatile memory332or the non-volatile memory334.

The program340may be stored in the memory330as software, and may include, for example, an operating system (OS)342, middleware344, or an application346.

The input module350may receive a command or data to be used by another component (e.g., the processor320) of the electronic device301, from the outside (e.g., a user) of the electronic device301. The input module350may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The sound output module355may output sound signals to the outside of the electronic device301. The sound output module355may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as playing multimedia or playing record. The receiver may be used for receiving incoming calls. According to an embodiment, the receiver may be implemented as separate from, or as part of the speaker.

The display module360may visually provide information to the outside (e.g., a user) of the electronic device301. The display module360may include, for example, a display, a hologram device, or a projector and control circuitry to control a corresponding one of the display, hologram device, and projector. According to an embodiment, the display module360may include a touch sensor adapted to detect a touch, or a pressure sensor adapted to measure the intensity of force incurred by the touch.

The audio module370may convert a sound into an electrical signal and vice versa. According to an embodiment, the audio module370may obtain the sound via the input module350, or output the sound via the sound output module355or a headphone of an external electronic device (e.g., an electronic device302) directly (e.g., wiredly) or wirelessly coupled with the electronic device301.

The interface377may support one or more specified protocols to be used for the electronic device301to be coupled with the external electronic device (e.g., the electronic device302) directly (e.g., wiredly) or wirelessly. According to an embodiment, the interface377may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

A connecting terminal378may include a connector via which the electronic device301may be physically connected with the external electronic device (e.g., the electronic device302). According to an embodiment, the connecting terminal378may include, for example, a HDMI connector, a USB connector, a SD card connector, or an audio connector (e.g., a headphone connector).

The camera module380may capture a still image or moving images. According to an embodiment, the camera module380may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module388may manage power supplied to the electronic device301. According to one embodiment, the power management module388may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

The battery389may supply power to at least one component of the electronic device301. According to an embodiment, the battery389may include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.

The communication module390may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device301and the external electronic device (e.g., the electronic device302, the electronic device304, or the server308) and performing communication via the established communication channel. The communication module390may include one or more communication processors that are operable independently from the processor320(e.g., the application processor (AP)) and supports a direct (e.g., wired) communication or a wireless communication. According to an embodiment, the communication module390may include a wireless communication module392(e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module394(e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate with the external electronic device via the first network398(e.g., a short-range communication network, such as Bluetooth™, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or the second network399(e.g., a long-range communication network, such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multi components (e.g., multi chips) separate from each other. The wireless communication module392may identify and authenticate the electronic device301in a communication network, such as the first network398or the second network399, using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module396.

The wireless communication module392may support a 5G network, after a 4G network, and next-generation communication technology, e.g., new radio (NR) access technology. The NR access technology may support enhanced mobile broadband (eMBB), massive machine type communications (mMTC), or ultra-reliable and low-latency communications (URLLC). The wireless communication module392may support a high-frequency band (e.g., the mmWave band) to achieve, e.g., a high data transmission rate. The wireless communication module392may support various technologies for securing performance on a high-frequency band, such as, e.g., beamforming, massive multiple-input and multiple-output (massive MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module392may support various requirements specified in the electronic device301, an external electronic device (e.g., the electronic device304), or a network system (e.g., the second network399). According to an embodiment, the wireless communication module392may support a peak data rate (e.g., 20 Gbps or more) for implementing eMBB, loss coverage (e.g., 164 dB or less) for implementing mMTC, or U-plane latency (e.g., 0.5 ms or less for each of downlink (DL) and uplink (UL), or a round trip of 1 ms or less) for implementing URLLC.

According to various embodiments, a processor320may perform a control related to an edge computing service using an edge device (e.g., the edge device210inFIG.2A) and a leaf device (e.g., the leaf device220inFIG.2A) in a home network by using an application346.

According to various embodiments, the application346may provide a function of registering an edge device and a leaf device in an IoT server (e.g., the IoT management server240inFIG.2A). For example, the processor320may use a communication module390to discover at least one edge device and at least one leaf device in a home network, and receive device information (e.g., identification information or network information) of each device.

The processor320may display a list of identified devices on the application346. The processor320may transmit, to the IoT server, device information of an edge device and a leaf device selected based on a user input, to request registration. In addition, the processor320may receive an input of the name and/or position information of each device on the application346.

According to various embodiments, in a state where an edge mode of an edge device is deactivated, the processor320may receive data (e.g., video streaming or sound data) obtained by a sensor of a leaf device and transmitted from the leaf device, from a cloud network. Thereafter, when an edge computing service is initiated, the data obtained by the leaf device is transmitted to the edge device, and a user device (e.g., the user device130inFIG.1, the user device230inFIG.2A, or the electronic device301) may directly receive sensor data and analysis data from the edge device or may receive same from the edge device via the cloud network (e.g., the cloud network140inFIG.1).

According to various embodiments, at least one of elements of an edge device or a leaf device may be configured to be at least partially identical to or similar to an element of the electronic device301(e.g., a user device). For example, a sensor of a leaf device may perform a function and/or operation substantially the same as that of a sensor module376of the electronic device301. In addition, a communication module (e.g., a communication module420inFIG.4) of an edge device may perform a function and/or operation substantially the same as that of a communication module390of the electronic device301, and an interface (e.g., the interface212inFIG.2A) may perform a function and/or operation substantially the same as that of an interface377of the electronic device301.

FIG.4is a block diagram illustrating an edge device according to various embodiments.

Referring toFIG.4, an edge device400according to various embodiments may include the communication module420, a processor410, and a memory430, and even though a part of the illustrated configuration is omitted or replaced, various embodiments disclosed herein may be implemented.

In an edge computing system according to various embodiments, multiple edge devices may exist, and each edge device may be a different type of device. For example, the edge device400may be implemented as a TV, a tablet PC, or a laptop PC. Hereinafter, a description of features for execution of, by the edge device400, a device unique function will be omitted, and only operations required for operation as the edge device400in an edge computing system will be described.

According to various embodiments, the communication module420may support wireless communication (e.g., short-range communication (Wi-Fi) or cellular communication), and transmit or receive data to or from a leaf device (e.g., the leaf device220inFIG.2A), a user device (e.g., the user device230inFIG.2A), and an external server (e.g., the IoT management server or the IoT hub server inFIG.2A). According to an embodiment, the communication module420may be connected to, as short-range communication, near field communication (NFC), Bluetooth™, Bluetooth™ low energy (BLE), WiFi Direct, a mesh network (e.g., Zigbee™ or z-wave), and/or ultra-wideband (UWB), and may be connected to a 4G network and/or a 5G network (e.g., standalone (SA) or non-standalone (NSA)) as cellular communication.

According to various embodiments, the memory430may include a volatile memory and/or a non-volatile memory, and may temporarily and/or permanently store various data used in at least one element (e.g., the processor410) of the edge device400. The memory430may store various instructions executable by the processor410. Such instructions may include various control commands including arithmetic and logical operations, data transfer, and input/output which is recognizable by the processor410.

Referring toFIG.4, the memory430may store device information, an edge runtime, a basic module, a device module, and a service module.

According to various embodiments, the device information may include identification information of the edge device400, and for example, include a device ID, a manufacturer, a model name, a capability, and a device type. In addition, the device information may further include at least a part of position information (e.g., a room or a living room) and network information (e.g., an IP address) of the edge device400. At least a part (e.g., position information) of the device information may be input by a user on an application of a user device (e.g., the electronic device301inFIG.3or the user device130inFIG.1).

According to various embodiments, the edge runtime may include a daemon program for interworking with an IoT server. The edge runtime is not executed in a state where an edge mode of the edge device400is deactivated, and, if a connection string is received from the IoT server (e.g., the IoT management server240inFIG.2A), may be executed using a value thereof.

According to various embodiments, the basic module is a program required for communication with the IoT server and may be configured as a container. The edge runtime and the basic module for edge computing may be installed in the edge device400through a software upgrade or a processing process of the edge device400. According to an embodiment, the edge device400may perform an over-the-air (OTA) software update as the software upgrade. For example, the OTA software update may include open mobile alliance (OMA) download, firmware OTA (FOTA), or plain FTP.

According to various embodiments, the edge device400may store the device module and the service module as modules required for an edge computing service. The device module may correspond to each connected leaf device, and the service module may correspond to each service to be performed. For example, multiple device modules and/or multiple service modules may be installed according to the number of connected leaf devices and the number of services to be performed (e.g., first device module219aand second device module219binFIG.2).

According to various embodiments, the device module may include programs supporting to convert data received from a leaf device according to a first protocol according to a second protocol (e.g., message queue telemetry transport (MQTT)) and transmit the converted data to an external server. According to an embodiment, the first protocol and the second protocol may be protocols related to a control channel for communication of the edge device400with the IoT server, and the first protocol may include a web socket. For example, the control channel is a channel for providing and/or processing device information, and the edge device400may use the control channel to communicate with the IoT server. In addition, the service module may include programs for processing and/or analyzing received data.

At the time of a software upgrade or a processing process of the edge device400, the edge runtime and the basic module may be installed without also installing the device module and the service module. Thereafter, when the edge device400is connected to a leaf device, the edge device may download and install the device module and the service module from the IoT server (e.g., the module registry254inFIG.2A).

The memory430may further store various programs for executing a unique function (e.g., a video output function of a TV) of the edge device400other than the illustrated configuration.

According to various embodiments, the processor410is an element which is capable of performing calculation or data processing related to control and/or communication of each element of the edge device400, and may be operatively, functionally, and/or electrically connected to each element of the edge device400, such as the communication module420or the memory430.

According to various embodiments, there may be no limit to calculation and data processing functions which is implementable by the processor410in the edge device400. However, hereinafter, operations for connecting, by the edge device400, to a leaf device for an edge computing service and receiving and installing a module required for the edge computing service will be described.

According to various embodiments, the processor410may register device information of the edge device400in the IoT server (e.g., the IoT management server240inFIG.2A). Here, the device information may include at least a part of identification information, position information, and network information of the edge device400. For example, the processor410may transmit device information stored in the memory430to a user device, the device information of the edge device400may be transmitted to the IoT server through an application of the user device, and the edge device400may be registered. If the device information of the edge device400is changed, the processor410may transmit updated information to the user device and/or the IoT server. In addition, for example, the edge device400may be in a state where an edge mode is activated and thus the edge runtime is operating (being executed). According to various embodiments, the processor410may receive a connection test command from the IoT server (e.g., the IoT management server240inFIG.2A) via the communication module420. The connection test command may include identification information and/or network information of a leaf device. The IoT server may search for at least one leaf device connectable with the edge device400among pre-registered several leaf devices, and transmit a connection test command for a leaf device selected by the user device to the edge device400.

According to various embodiments, the processor410may use the communication module420to identify at least one external device located on the same network as that of the electronic device, and may transmit identification information of the at least one external device to the IoT server. In addition, when a particular leaf device is selected by the user device among the at least one external device, the IoT server may transmit a connection test command for the corresponding leaf device to the edge device400.

According to various embodiments, the processor410may identify whether a connection to the leaf device is possible, by using the communication module420, in response to reception of a connection test command. The processor410may transmit a connection test result to the IoT server.

According to various embodiments, the processor410may receive at least one module (e.g., a device module or a service module) for performing an edge computing service from the IoT server (e.g., the IoT hub server250inFIG.2A) via the communication module420, and store and install the received module in the memory430. The IoT server may identify a performable edge computing service, based on device information (e.g., a device ID, a manufacturer, a model name, a capability, and a device type of the edge device400) of the edge device400and device information (e.g., a device ID, a manufacturer, a model name, a capability, and a device type of a leaf device) of a leaf device, and provide a module required for performing the corresponding service to the processor410.

For example, if the leaf device is an IP camera, the IoT server may provide an audio video module (AV module) required for receiving video streaming transmitted from the leaf device and transmitting same to a cloud network, and an artificial intelligence vision module (AI vision module) that performs video analysis in an artificial intelligence vision (AI vision) service. Alternatively, if the leaf device is an AI speaker, the IoT server may provide a sound module required for required for receiving sound data transmitted from the leaf device and transmitting same to the cloud network, and an AI sound analysis module that analyzes the received sound data.

As described above, when a leaf device is connected, modules required for an edge computing service may be installed in the edge device400, thereby dynamically implementing the edge computing service by using an idle computing resource of the edge device400. According to an embodiment, an edge device and a leaf device may generate a third channel (e.g., a control channel) for processing control information (e.g., control information, event information, or device information) and a fourth channel (e.g., a streaming channel) for processing (e.g., transferring) video information (e.g., video streaming). In addition, when the edge device and the leaf device are connected to each other, video information may be transferred to the edge device by using the fourth channel rather than a second channel (e.g., streaming channel) between a first channel (e.g., a control channel) and the second channel (streaming channel) between the leaf device and the IoT server, which are existing channels. According to various embodiments, the processor410may execute the stored device module and service module, and perform an edge computing service, based on data received from the leaf device. For example, if the leaf device is an IP camera, the edge device400may provide various video analysis-based services, such as facial recognition, or user recognition and tracking, by analyzing a received video by using the AI vision module. Alternatively, if the leaf device is an AI speaker, the edge device400may perform a service, such as pet monitoring, window damage detection, or alarm sound detection, by analyzing sound data by using the AI sound analysis module.

The processor410may transmit sensor data received from the leaf device and analysis data generated by the service module to the IoT server in real time by using the communication module420. According to another embodiment, the edge device400may transfer information (e.g., video information, sensor data, or analysis data) obtained by the leaf device, to the user device without using the IoT server.

According to various embodiments, the processor410may disconnect two devices therebetween if the leaf device disappears from a home network (e.g., disconnection from AP). In this case, the processor410may report disconnection between two devices to the IoT server. When two devices are disconnected from each other, the leaf device may access the cloud network again and a service may be continuously provided. For example, the leaf device may use the first channel (e.g., control channel) and the second channel (e.g., streaming channel) generated between the leaf device and the IoT server, to provide a service of the leaf device to the user device.

FIG.5is a block diagram illustrating a leaf device according to various embodiments.

Referring toFIG.5, a leaf device500according to various embodiments may include the communication module520, a processor510, and a sensor540, and even though a part of the illustrated configuration is omitted or replaced, various embodiments disclosed herein may be implemented.

In an edge computing system according to various embodiments, multiple leaf devices500may exist, and each leaf device500may be a different type of device. For example, the leaf device may be implemented as a camera, a refrigerator, a bulb, or a digital thermometer. Hereinafter, a description of features for execution of, by the leaf device500, a device unique function will be omitted, and only operations required for operation as the leaf device500in an edge computing system will be described.

According to various embodiments, the communication module520may support wireless communication (e.g., Wi-Fi or cellular communication), and transmit or receive data to or from an edge device (e.g., the edge device210inFIG.2Aor the edge device400inFIG.4), a user device (e.g., the user device230inFIG.2Aor the user device301inFIG.3), and an external server (e.g., the IoT management server240or the IoT hub server250inFIG.2A).

According to various embodiments, the leaf device500may include the at least one sensor540. For example, if the leaf device500is an IP camera, the leaf device may include an image sensor that obtains surrounding video data, and if the leaf device500is an AI speaker, the leaf device may include a microphone that detects surrounding sound. Alternatively, the leaf device500may include the sensor540for detecting various data, such as temperature, pressure, or impact amount, according to the type thereof.

According to various embodiments, the processor510is an element which is capable of performing calculation or data processing related to control and/or communication of each element of the leaf device500, and may be operatively, functionally, and/or electrically connected to each element of the leaf device500, such as the communication module520or the sensor540.

According to various embodiments, the processor510may register device information of the leaf device500in the IoT server. Here, the device information may include at least a part of identification information, position information, and network information of an edge device. For example, the processor510may transmit device information stored in the memory to a user device, the device information of the leaf device500may be transmitted to the IoT server through an application of the user device, and the leaf device500may be registered.

According to various embodiments, in a state where the leaf device500is not connected to an edge device, the processor510may transmit data (e.g., video streaming) obtained via the sensor540, to the IoT server. According to various embodiments, the leaf device500may establish a first channel (e.g., control channel) and a second channel (e.g., streaming channel) with the IoT server, the leaf device500and the IoT server may transmit or receive control information and/or event information through the first channel, and video streaming obtained by the leaf device500may be transmitted to the IoT server (or a cloud network) through the second channel.

According to various embodiments, in response to a connection test command received from the IoT server, the processor510may attempt to connect an edge device via the communication module520and transmit a test result to the IoT server. Thereafter, the processor510may establish a connection with an edge device via the communication module520.

For example, in response to a connection test command received from the IoT server, the processor510may broadcast, on a local network, data (e.g., video streaming) sensed and transmitted by the leaf device500and a signal including a service performable using the leaf device500. At least partially at the same time, the edge device may broadcast, on the local network, a signal including an edge computing service performable by the edge device. The processor510may identify which service is providable by the edge device, from the signal broadcast from the edge device, then identify whether an edge computing service for data of the leaf device500is possible, and attempt to connect the edge device.

According to various embodiments, if the leaf device500is connected to the edge device and an edge mode of the edge device is activated, the leaf device500may transmit sensor data to the edge device through the established connection. In this case, the edge device and the leaf device500may establish a third channel (e.g., control channel) for transmitting or receiving control information and/or event information, and establish a fourth channel (e.g., streaming channel) for transmitting or receiving video streaming. While video streaming is transmitted to the edge device through edge computing, the second channel for transmission or reception of sensor data between the leaf device500and the IoT server need not be used.

According to various embodiments, if the processor510is disconnected from the edge device, the processor may transmit sensor data to the IoT server again.

FIG.6is a component diagram illustrating interactions between devices of an edge computing system according to various embodiments.

InFIG.6, an IoT management server640(e.g., the IoT management server240) and an IoT hub server650(e.g., the IoT hub server250inFIG.2A) and inFIG.2A) are illustrated as a device on a cloud network. However, a function of the IoT management server640and the IoT hub server650may be implemented by one server device (e.g., the IoT server260inFIG.2B) or may be dispersedly processed by three or more devices. An edge device610and a leaf device620may be connected to each other on the same home network.

According to various embodiments, in operation601, the leaf device620(e.g., the leaf device220inFIG.2Aor the leaf device500inFIG.5) may register device information of the leaf device620in the IoT management server640(e.g., SmartThings™ server). The device information of the leaf device620may include at least a part of identification information (a device ID, a manufacturer, a model name, a capability, and a device type), position information (e.g., a room or a living room) and network information (e.g., an IP address) of the leaf device620. According to an embodiment, there may be various methods for connecting to the IoT server according to the device information of the leaf device620. For example, if the leaf device620is an IP-based (IP address) device, the leaf device may connect to the IoT server (e.g., the IoT management server640) by using a service set identifier (SSID), and if the leaf device is not an IP-based device (e.g., BLE, Zigbee™, or z-wave), the leaf device may connect to the IoT server (e.g., the IoT management server640) via a hub device (e.g., the hub device124inFIG.1) or by using a user device (e.g., the user device230inFIG.2Aor the user device301inFIG.3) as a hub device. According to various embodiments, in operation602, the edge device610(e.g., the edge device210inFIG.2Aor the edge device400inFIG.4) may register device information of the edge device610in the IoT management server640. The device information of the edge device610may include at least a part of identification information (a device ID, a manufacturer, a model name, a capability, and a device type), position information (e.g., a room or a living room) and network information (e.g., an IP address) of the edge device610. For example, the edge device610may transmit the device information of the edge device610to the IoT management server640in a deactivated state of an edge mode. For example, in a state where an edge runtime is not executed, the edge device610may execute a basic module (e.g., a container of a program required for communication with the IoT management server640) for edge computing to transmit the device information. In this embodiment, the edge device610installs an edge runtime and a basic module for edge computing, but a module (e.g., the device module219or the service module218inFIG.2A) for executing an edge computing service for data transmitted from the leaf device620might not have been installed yet.

According to various embodiments, the IoT management server640may store pieces of information received from the at least one leaf device620and/or the edge device610in a memory (e.g., a database).

According to various embodiments, operations601and602may be collectively referred to as a process of registration in an IoT server device.

According to various embodiments, in operation602′, a user may select execution of an edge mode of the edge device610on an application of a user device (e.g., the user device230inFIG.2Aor the user device301inFIG.3). The application of the user device may transmit information on the selected edge device610and/or leaf device620to the IoT management server640. According to various embodiments, in operation603, the IoT management server640(e.g., the provision manager242inFIG.2A) may transmit a connection string to the edge device610to be connected to the IoT hub server650. According to various embodiments, in operation604, the edge device610may receive an issued connection string and use a value thereof to execute the edge runtime and activate an edge mode. The edge device610may be connected to the IoT hub server650by execution of the edge runtime. For example, the connection string is an identification (ID) for interworking with the IoT hub server650and may be data allowing identification of the IoT hub server650. According to various embodiments, operations602′,603, and604may be collectively referred to as a process of installation of an edge runtime.

According to various embodiments, in operation605, the edge device610and the leaf device620may each identify (discover) whether a device capable of edge computing exists on the home network. For example, the edge device610may identify the leaf devices620accessing the same access point (AP). According to various embodiments, the edge device610and the leaf device620may identify (discover) whether there is a device capable of edge computing in a device-to-device (D2D) scheme rather than an IP-based AP device. For example, the edge device610and the leaf device620may perform a discovery process (e.g., operation605and operation605′) through establishment of communication such as BLE or WiFi Direct.

According to various embodiments, in operation605′, the edge device610and/or the leaf device620may transmit information of a device connectable on the home network to the IoT management server640. According to another embodiment, the IoT management server640may search for the leaf device620connectable with the edge device610, based on account and/or position information from device information of registered devices, and transfer same to the edge device610and/or the leaf device620.

According to various embodiments, when a connection between the edge device610and the leaf device620is established, the IoT management server640(e.g., the edge-leaf manager246inFIG.2A) may identify a service available via the edge device610and the leaf device620. For example, if the leaf device620is an IP camera, it may be identified that an AI vision service including video analysis is possible.

According to various embodiments, the IoT management server640may identify at least one module required by the edge device610for the identified edge computing service. The module for an edge computing service may include a device module supporting to receive data transmitted from the leaf device620and transmit same to the cloud network, and a service module including programs for performing the edge computing service, based on the received data. For example, the IoT management server640may identify an AV module required for the edge device610to transmit video streaming transmitted from the leaf device620, which is an IP camera, to the cloud network, and an AI vision module that performs video analysis in an AI vision service. The IoT management server640may store information of modules required for each edge computing service, as described above, in a database. According to various embodiments, operations605and605′ may be referred to as a discovery process.

According to yet another embodiment, the process of installation of the edge runtime according to operations602′,603, and604may be executed after the discovery process of operations605and605′ unlike the illustrated embodiment.

According to various embodiments, the edge device610and the leaf device620may be connected to each other through an authentication process. When the edge device610and the leaf device620are connected to each other, connection between two devices may be updated in the IoT management server640(e.g., the edge-leaf manager inFIG.2A).

According to various embodiments, in operations606and607, the IoT management server640(e.g., the module manager inFIG.2A) may transmit information of a module required for an edge computing service (e.g., AI vision service) to the IoT hub server650and/or the edge device610. The edge device610may receive the information of the required module, and the edge runtime may download the corresponding module from the IoT hub server650(e.g., the module registry254inFIG.2A) in the edge runtime and install same on the edge device610.

According to various embodiments, when the installed module (e.g., a device module or a service module) is updated, the edge device610may download and install a new module from the IoT hub server650(e.g., the module registry254inFIG.2A). According to various embodiments, when the existing edge device610performs an image classification function using data received via the leaf device620(e.g., IP camera), the new edge device610may perform an image analysis function related to image classification and additional functions (e.g., facial recognition), based on an update operation.

According to various embodiments, in a state where the edge device610is connected to the leaf device620, the edge device may always execute a device module corresponding to the corresponding leaf device620in order to receive data from the leaf device620. The device module may receive data from the leaf device620via a first protocol, and transmit data to the IoT server via a second protocol.

The leaf device620may transmit obtained data (e.g., video streaming) to the cloud network (e.g., the IoT management server640) before being connected to the edge device610, and may transmit data and events to the edge device610in a state where the leaf device is connected to the edge device610. The edge device610may transmit at least a part of the data and events received from the leaf device620, to the cloud network (e.g., the IoT management server640).

According to various embodiments, the two devices may be disconnected from each other if the edge device610and/or the leaf device620disappear from the home network (e.g., disconnection from AP). In this case, disconnection between the two devices may be updated in the IoT management server640(e.g., the edge-leaf manager246inFIG.2A). When the two devices are disconnected from each other, the leaf device620may access the cloud network again and a service may be continuously provided.

FIG.7is a sequence diagram illustrating an edge computing service method according to various embodiments.

Referring toFIG.7, an edge computing system may include a user device730, a leaf device720, an edge device710, and an IoT server740. The IoT server740may include multiple server devices (e.g., the IoT management server240and the IoT hub server250inFIG.2A) existing on a cloud network, and operations thereof may be dispersedly processed by several server devices.

According to various embodiments, in operation751, the user device730may identify device information of the leaf device720, and register the device information of the leaf device720in the IoT server740. The user device730may identify the device information of the leaf device720on a home network through a discovery process, and the device information of the leaf device720may include at least a part of identification information (a device ID, a manufacturer, a model name, a capability, and a device type), position information (e.g., a room or a living room) and network information (e.g., an IP address) of the leaf device720. If a corresponding device is the leaf device720is checked on an application of the user device730, registration of the corresponding device in the IoT server740, as the leaf device720, may be requested. In addition, position information (e.g., room or living room) of the leaf device720may be registered on the application of the user device730, and then be transmitted to the IoT server740.

According to various embodiments, in operation752, the user device730may identify device information of the edge device710, and register the device information of the edge device710in the IoT server740. The user device730may identify the device information of the edge device710on the home network through a discovery process, and the device information of the edge device710may include at least a part of identification information (a device ID, a manufacturer, a model name, a capability, and a device type), position information (e.g., a room or a living room) and network information (e.g., an IP address) of the edge device710. When a corresponding device is the edge device710is checked on the application of the user device730, registration of the corresponding device in the IoT server740, as the edge device710, may be requested. In addition, position information (e.g., room or living room) of the edge device710may be registered on the application of the user device730, and then be transmitted to the IoT server740. In a process of registering the edge device710, an edge mode of the edge device710need not be activated.

According to various embodiments, pieces of information received from the at least one leaf device720and the edge device710may be stored in a database, and a registration account and position information of each device and whether each device is the leaf device720or the edge device710may be stored.

According to various embodiments, in operation761, the edge device710may execute an edge runtime and activate an edge mode. The edge device710may receive a connection string issued from the IoT server740and use a value thereof to execute the edge runtime.

According to various embodiments, in operation762, the IoT server740may search for at least one leaf device720connectable with the edge device710among a plurality of pre-registered leaf devices720. If the user device730selects a particular edge device710to perform an edge computing service, the IoT server740may identify a user account, device information and position information of the selected edge device710, and a local network, and may search for leaf devices720connectable with the edge device710on the database, based on the corresponding pieces of information.

According to various embodiments, in operation763, the IoT server740may provide a list of leaf devices720connectable with the edge device710to the user device730.

According to various embodiments, in operation764, the user device730may provide the received list through the application, select the leaf device720to be connected to the edge device710on the list according to a user input, and transmit same to the IoT server740.

According to an embodiment, to enable operations763and764, the user device730may provide a user interface (e.g.,FIG.9AtoFIG.9C) related to an operation of selecting a leaf device. According to another embodiment, the user device730may provide a user interface (e.g.,FIG.10AtoFIG.10C) related to an operation of selecting an edge device, based on a user input obtained from the user device730.

According to various embodiments, in operation765and operation766, the IoT server740may transmit a connection test command to the edge device710and the leaf device720. For example, the IoT server740may transmit identification information and/or network information of a counterpart device to request identification of whether a connection to the counterpart device is possible.

According to various embodiments, in operation767and operation768, the edge device710and/or the leaf device720may attempt to connect to each other on the home network, and transmit a test result to the IoT server740. The IoT server740may transmit a connection test result of the edge device710and the leaf device720to the user device730, and the user device730may provide a user interface related to the connection test result. For example, the user interface of the user device730may provide an interface screen relating to a connection attempt process, such as connection request, connection failure, or establishment of a connectivity channel on the home network by the edge device710and the leaf device720, and/or, if a message is mutually transmitted or received through a connectivity channel and a process of authentication on whether a device is a reliable device is completed, may provide information related to success or failure of the authentication.

According to various embodiments, in operation771, if the edge device710and the leaf device720are identified as being connectable with each other, the IoT server740may identify at least one module required for an edge computing service. For example, the IoT server740may identify a performable edge computing service, based on device information (e.g., a device ID, a manufacturer, a model name, a capability, and a device type of the edge device710) of the edge device710and device information (e.g., a device ID, a manufacturer, a model name, a capability, and a device type of the leaf device720) of the leaf device720.

According to various embodiments, the IoT server740may match performable services with device information of several devices and store same in the database. In addition, the IoT server740may store a module required for each service and modules required for transmitting and/or processing data of a particular device.

According to various embodiments, the at least one module for an edge computing service may include a device module supporting to receive data transmitted from the leaf device720and transmit same to the cloud network, and a service module including programs for performing the edge computing service, based on the received data. For example, the device module may include programs supporting to convert data received from the leaf device720according to a first protocol according to a second protocol (e.g., message queue telemetry transport (MQTT)) and transmit the converted data to an external server. In addition, the service module may include programs for processing and/or analyzing received data.

For example, if the leaf device720is an IP camera, the IoT server740may identify an AV module required for receiving video streaming transmitted from the leaf device720, and transmitting same to the cloud network, and an AI vision module that performs video analysis in an AI vision service. Alternatively, if the leaf device720is an AI speaker, the IoT server740may identify a sound module required for required for receiving sound data transmitted from the leaf device720and transmitting same to the cloud network, and an AI sound analysis module that analyzes the received sound data.

According to various embodiments, in operation772, the IoT server740may transmit an identified module (e.g., AV module and AI vision module) to the edge device710.

According to various embodiments, in operation773, the edge device710may install and execute received modules.

According to various embodiments, in operation781, the edge device710and the leaf device720may establish a connection therebetween through mutual authentication.

According to various embodiments, in operation782, the leaf device720may transmit data (e.g., video streaming) obtained using a sensor to the edge device710by using the established connection.

According to various embodiments, in operation783, the edge device710may analyze data received from the leaf device720. For example, if the leaf device720is an IP camera, the edge device710may provide various video analysis-based services, such as facial recognition, or user recognition and tracking, by analyzing a received video by using the AI vision module. Alternatively, if the leaf device720is an AI speaker, the edge device710may perform a service, such as pet monitoring, window damage detection, or alarm sound detection, by analyzing sound data by using the AI sound analysis module.

According to various embodiments, in operation784, the edge device710may transmit the data received from the leaf device720and the data analyzed by the edge device710to the IoT server740in real time.

According to various embodiments, in operation785, the IoT server740may transmit sensor data and analysis data received from the edge device710to the user device730. The user device730may provide the received data (e.g., video streaming or an alarm) to a user through the application.

FIG.8AandFIG.8Bare component diagrams illustrating interactions between devices during operation of an edge computing service method according to various embodiments.

FIG.8Aillustrates an operation of a state where a leaf device820(e.g., the leaf device220inFIG.2Aor the leaf device500inFIG.5) is connected to a cloud network before the leaf device820and an edge device810(e.g., the edge device210inFIG.2Aor the edge device400inFIG.4) are connected.

Referring toFIG.8A, a TV positioned on a home network may operate as the edge device810, and an IP camera may be newly installed as the leaf device820. The edge device810may be pre-registered in an IoT server840(e.g., the IoT management server240inFIG.2A) by transmitting device information thereto.

A user may newly purchase and install indoors the leaf device820to be connected to the home network via an AP. The leaf device820may be registered in the IoT server840by transmitting device information of the leaf device820thereto.

Referring toFIG.8A, before the leaf device820is connected to the edge device810after being registered, the leaf device820may transmit data (e.g., video streaming) obtained via a sensor (e.g., the sensor540inFIG.5) to the IoT server840. According to various embodiments, the leaf device820may establish a first channel851and a second channel852with the IoT server840, the leaf device820and the IoT server840may transmit or receive control information and/or event information through the first channel851, and video streaming obtained by the leaf device820may be transmitted (in a first transmission stage) to the IoT server840(or a cloud network) through the second channel852. The IoT server840may transmit the obtained video streaming (in a second transmission stage) to a user device830. The edge device810has not been connected to the leaf device820yet, and the edge device810and the IoT server840may establish a channel for transmitting or receiving control information and/or event information.

According to various embodiments, the leaf device820and the edge device810may broadcast, on a local network, a signal for notifying that each device itself is a device capable of edge computing. If the leaf device820identifies a TV that is the edge device810, the leaf device820may connect to the IoT server840(e.g., the edge-leaf manager246inFIG.2Aor the edge-leaf manager265inFIG.2B) of the edge device810and request same to identify providable services. The IoT server840(e.g., the edge-leaf manager246and the module manager244inFIG.2A) may identify, based on device information of the leaf device820and the edge device810, a service available between the leaf device820and the edge device810, and at least one module required for the edge device810to perform the corresponding service.

According to various embodiments, the IoT server840may provide the module required for performing the identified service to the edge device810. For example, the IoT server840may provide, to the edge device810, an audio/video (AV) module which routes video streaming of the leaf device820to the IoT server840(or cloud network) and functions to communication between the edge device810and the IoT server840, and an AI vision module which performs video analysis.

FIG.8Billustrates an operation after the leaf device820and the edge device810are connected to each other.

According to various embodiments, the edge device810and the leaf device820may establish a connection therebetween on the home network, and the edge device810may execute an installed module. The leaf device820may transmit video streaming (in a first transmission stage) to the edge device810through the established connection, and the edge device810may transmit the received video streaming (in a second transmission stage) to the IoT server840. The edge device810may receive, via the AV module, video streaming and control information and/or event information received from the leaf device820, and the AI vision module may analyze the received video to provide various video analysis-based services, such as facial recognition, or user recognition and tracking. Video streaming and analyzed data may be transmitted (in a third transmission stage) to the user device830via the IoT server840in real time.

According to another embodiment, the edge device810may establish a direct connection with the user device830, and directly transmit data to the user device830without going through the IoT server840. For example, the edge device810may directly transmit video streaming date and data generated by analyzing (e.g., video information-based user recognition, marking, or behavior tracking) video streaming received from the leaf device820, through a connection established with the user device830.

According to various embodiments, the edge device810and the leaf device820may establish a third channel861for transmitting or receiving control information and/or event information, and establish a fourth channel862for transmitting or receiving video streaming. Referring toFIG.8B, while video streaming is transmitted to the edge device810through edge computing, the second channel852for transmission or reception of video streaming between the leaf device820and the IoT server840need not be used.

According to various embodiments, in a state where the leaf device820(e.g., IP camera) and the edge device810(e.g., TV) are connected and perform an edge computing service, a new leaf device820(e.g., AI speaker) may be connected.

The AI speaker may broadcast, on the local network, that the AI speaker itself is capable of edge computing. When the AI speaker discovers a TV that is an edge device, the AI speaker may connect to the IoT server840(e.g., the edge-leaf manager and the module manager inFIG.2A) to identify providable services. For example, the AI speaker may transmit a connection request to the TV, and the TV may be connected to the AI speaker through an authentication process.

After the AI speaker is connected to the TV, the TV may install, from the IoT server840, a module related to a service to be provided to the AI speaker. For example, the TV may download, from the IoT server840(e.g., the module registry254inFIG.2A), and install a sound module for receiving sound data input in a microphone of the AI speaker, and an AI sound analysis module that analyzes the received sound data.

The TV may execute the installed module and perform an edge computing service. The AI speaker may detect surrounding sound and transfer sound data to the sound module of the TV, and the AI sound analysis module may analyze the received sound data, and transfer the result thereof to the IoT server840via the sound module. According to the analysis of sound data as described above, a service, such as pet monitoring, window damage detection, or alarm sound detection, may be performed.

According to various embodiments, an edge device may provide various edge computing services according to data transmitted from a leaf device, and an IoT server may store modules required for each edge computing service.

An example of an edge computing service available with respect to a leaf device and a device module and a service module required accordingly is as follows.

In [Table 1] above, a message queueing telemetry transport (MQTT) broker may be a module that provides a channel through which a command and/or an event are exchanged between a leaf device and an edge device. For example, an MQTT protocol may include a broker, a publisher, and a subscriber. If an MQTT broker is installed in the edge device810, the MQTT broker of the edge device may perform a function of relaying a topic issued in the publisher to the subscriber.

An audio video (AV) stream gateway may perform a function of relaying audio and/or video streaming transmitted from a leaf device, to an artificial intelligence (AI) module.

FIG.9AtoFIG.9CandFIG.10AtoFIG.10Cillustrate application interface screens of a user device according to various embodiments.

FIG.9AtoFIG.9Cillustrate a user interface of an application for selecting a leaf device (e.g., the leaf device220inFIG.2Aor the leaf device500inFIG.5) to be connected to an edge device in a user device900.

Referring toFIG.9A, the user device900may provide a user interface (UI) for, when a particular edge device is initially registered or reconfigured, initiating a search for a leaf device to be connected to the edge device. According to various embodiments, when a leaf device search menu910is selected on the UI, the user device900may search for at least one leaf device positioned on a home network (e.g., connected to the same AP) by using a communication module (e.g., the communication module390inFIG.3).

Referring toFIG.9B, the user device900may display lists921and922of at least one leaf device identified on the home network. According to various embodiments, the user device900may display leaf devices to be distinguished according to each position (e.g., living room or kitchen) of the leaf device. According to various embodiments, the UI of the user device900is not limited to the illustrated example, and may display leaf devices to be distinguished according to each type (e.g., camera, speaker, or bulb) of the leaf device, or display leaf devices, based on multiple criteria (e.g., type, network registration date, and distance to the user device). The user device900may receive an input of selection of a leaf device to be connected to the edge device among the displayed at least one leaf device. The user device900may transmit device information of the selected leaf device to an IoT server via the communication module.

If a leaf device to be connected to the edge device is selected by the user device900, the IoT server may transmit a connection test command to the corresponding edge device and leaf device. The edge device and/or leaf device may transmit a connection test result to the IoT server, and the IoT server may transmit the connection test result to the user device900.

Referring toFIG.9C, the user device900may display a UI with indicators (e.g., connectable indicator931and unconnectable indicator932) showing whether the selected edge device and leaf device are connectable to each other.

For example, when the edge device and the leaf device are connected to each other, data (e.g., video streaming) obtained by the leaf device may be transmitted to the edge device, and the edge device may transmit data received from the leaf device to the IoT server. The user device900may receive the data of the leaf device from the IoT server and display same on the application. According to various embodiments, the edge device may encrypt data (e.g., video) received from the leaf device and transmit the encrypted data to the IoT server, and the user device900may receive decoded data of the leaf device, based on a designated key (e.g., a password or an authentication key).

According to various embodiments, when data between the leaf device and the edge device is transmitted to the IoT server, the data may be transferred after being processed in a different type. For example, the edge device may obtain video data from the leaf device, and perform a user recognition function, based on the obtained video data. For example, the edge device may transfer alarm information (e.g., processed data of the leaf device) related to a recognized user to the IoT server and/or the user device900, based on user recognition, and the user device900may obtain the alarm information from the IoT server. For example, data obtained from a sensor of the leaf device may be stored on the home network on which the leaf device and the edge device exist, and the IoT server and/or the user device900may receive information processed in a different type (e.g., alarm information) from the edge device.

FIG.10AtoFIG.10Cillustrate a user interface of an application for selecting an edge device to be connected to a leaf device in a user device. According to various embodiments, the user interface of the application for selecting the edge device may be configured to be at least partially similar to a user interface (e.g.,FIG.9AtoFIG.9C) of an application for selecting a leaf device.

Referring toFIG.10A, a user device1000may provide a user interface (UI) for, when a particular leaf device is initially registered or reconfigured, initiating a search for an edge device to be connected to the leaf device. According to various embodiments, when an edge device search menu1010is selected on the UI, the user device1000may search for at least one edge device positioned on a home network (e.g., connected to the same AP) by using a communication module (e.g., the communication module420inFIG.4). For example, the user device may search for a device in which an edge runtime is installed, among several electronic devices of the home network.

Referring toFIG.10B, the user device1000may display lists1021and1022of at least one edge device identified on the home network. According to various embodiments, the user device1000may display edge devices to be distinguished according to each position (e.g., living room or kitchen) of the edge device. A user may select one edge device connected to the leaf device on the displayed lists.

According to various embodiments, an IoT server may store a user account and/or position information of an edge device and a leaf device in a database, and the user device1000may display a list of leaf devices or edge devices registered to have the same user account and/or position as that of a corresponding edge device or leaf device when searching for a surrounding device.

Referring toFIG.10C, the user device1000may display a UI with indicators (e.g., connectable indicator1031and unconnectable indicator1032) showing whether the selected edge device and leaf device are connectable to each other.

For example, when the edge device and the leaf device are connected to each other, data (e.g., video streaming) obtained by the leaf device may be transmitted to the edge device, and the edge device may transmit data received from the leaf device to the IoT server. The user device1000may receive the data of the leaf device from the IoT server and display same on the application.

According to various embodiments, the IoT server may provide a designated leaf device and/or edge device rather than the lists921and922of at least one leaf device inFIG.9Band/or the lists1021and1022of at least one edge device inFIG.10B. For example, in a case of an operation related to searching for a leaf device, the IoT server may determine an optimal leaf device from the database of the IoT server, and provide the determined leaf device to the user device900. In addition, in a case of an operation related to searching for an edge device, the IoT server may determine an optimal edge device from the database of the IoT server, and provide the determined edge device to the user device1000. According to various embodiments, an operation of determining an optimal leaf device and/or edge device may be performed based on at least one of a network registration date or numerical value information (e.g., ranking) on a hardware condition (e.g., CPU, GPU, or memory) of a leaf device and/or an edge device.

FIG.11is a flowchart illustrating an edge computing service method of an edge device according to various embodiments.

The illustrated method may be performed by an edge device (e.g., the edge device210inFIG.2A, the edge device400inFIG.4, the edge device610inFIG.6, or the edge device710inFIG.7) described above, and a technical feature described above will not be described hereinafter.

In operation1110, an edge device may receive a connection test command for a leaf device from an external server (e.g., the IoT management server240inFIG.2A). The connection test command may include identification information and/or network information of the leaf device. The external server may search for at least one leaf device connectable with the edge device among pre-registered several leaf devices and transmit a connection test command for a leaf device selected by a user device to the edge device.

In operation1120, in response to the reception of the connection test command, the edge device may identify whether a connection to the leaf device is possible, by using a communication module, and transmit a result of the identification to the external server.

In operation1130, when a connection test result indicates that the leaf device is connectable, the edge device may receive at least one module for performing an edge computing service from the external server, and store and install same in a memory. The external server may identify a performable edge computing service, based on device information of the leaf device and device information of the edge device, for example, identification information (e.g., a device ID, a manufacturer, a model name, a capability, and a device type), position information (e.g., room or living room), and network information (e.g., IP address), and may provide a module required for performing the corresponding service to the edge device.

For example, if the leaf device an IP camera, the external server may provide an AV module required for receiving video streaming transmitted from the leaf device, and transmitting same to a cloud network, and an AI vision module that performs video analysis in an AI vision service. Alternatively, if the leaf device is an AI speaker, the external server may provide a sound module required for required for receiving sound data transmitted from the leaf device and transmitting same to the cloud network, and an AI sound analysis module that analyzes the received sound data.

In operation1140, the edge device may execute the installed at least one module, and may perform the edge computing service, based on data received from the leaf device. For example, if the leaf device is an IP camera, the edge device may provide various video analysis-based services, such as facial recognition, or user recognition and tracking, by analyzing a received video by using the AI vision module. Alternatively, if the leaf device is an AI speaker, the edge device may perform a service, such as pet monitoring, window damage detection, or alarm sound detection, by analyzing sound data by using the AI sound analysis module. A processor may transmit sensor data received from the leaf device and analysis data generated by the service module to a cloud network in real time by using a communication module.

FIG.12is a flowchart illustrating an edge computing support method of an IoT server according to various embodiments.

The illustrated method may be performed by an IoT server (e.g., the IoT server740inFIG.7) described above, and a technical feature described above will not be described hereinafter. The illustrated method may be performed by one IoT server, and some of illustrated operations may be dispersedly processed by multiple server devices (e.g., the IoT management server240and the IoT hub server250inFIG.2A).

In operation1210, an IoT server may receive selection of an edge device and a leaf device from a user device. The user device may select an edge device and a leaf device to connect to each other for an edge computing service on a user interface of an application, and transmit same to the IoT server.

In operation1220, the IoT server may transmit a test command to identify whether the edge device and the leaf device are connectable. The edge device and/or the leaf device may attempt to wirelessly connect to a counterpart device, and then notify the IoT server of a result thereof.

In operation1230, if the edge device and the leaf device are identified as being connectable, the IoT server may determine at least one module required for performing an edge computing service, based on device information of the edge device and the leaf device. For example, if the leaf device an IP camera, the IoT server may provide an AV module required for receiving video streaming transmitted from the leaf device, and transmitting same to a cloud network, and an AI vision module that performs video analysis in an AI vision service. Alternatively, if the leaf device is an AI speaker, the IoT server may provide a sound module required for required for receiving sound data transmitted from the leaf device and transmitting same to the cloud network, and an AI sound analysis module that analyzes the received sound data.

In operation1240, the IoT server may transmit at least one module stored in the IoT server to the edge device. Alternatively, if a module required for an edge computing service is stored in a different server device (e.g., IoT hub server) as illustrated inFIG.2A, the IoT server may request the IoT hub server to transmit at least one module to the edge device.

FIG.13illustrates an AI computing distributed processing platform according to various embodiments.

Referring toFIG.13, an AI computing distributed processing platform1300may include a leaf device1320, an edge device1310, and a cloud network1340(e.g., the IoT management server240and the IoT hub server250inFIG.2Aor the IoT server260inFIG.2B). According to various embodiments, the edge device1310may include not only a TV (e.g., the TV112inFIG.1) or a table PC (e.g., the tablet PC111inFIG.1) but also consumer premises equipment (CPE) of a fixed wireless access (FWA) type of 4G and 5G communication.

According to various embodiments, the AI computing distributed processing platform1300may dispersedly perform AI computing processing related to data obtained by the leaf device1320, based on sensitivity of the data and/or a computing resource of each device.

For example, if the leaf device is not connected to the edge device or the cloud network, on-device AI computing processing may be performed using a hardware and/or software resource of the leaf device. If the leaf device1320is connected to the edge device1310and the edge device1310is able to execute a module for AI computing processing, data having a high sensitivity among pieces of data generated in the leaf device1320may be processed by AI computing (edge AI computing) on the edge device1310, and data (e.g., general data) having a low sensitivity may be transmitted to the cloud network1340via the edge device1310and then be processed by AI computing (cloud AI computing) on a cloud. Alternatively, if the leaf device1320includes hardware capable of AI computing processing, some of AI computing may be dispersedly processed by on-device AI computing on the leaf device1320.

FIG.14is a flowchart illustrating an AI computing distributed processing method according to various embodiments.

The illustrated method may be performed by the leaf device, the edge device, and the cloud network (e.g., the IoT management server and the IoT hub server inFIG.2Aor the IoT server inFIG.2B).

In this embodiment, an edge device is a TV, and may execute an edge runtime in addition to a unique function of video output. An edge device may be registered in an IoT server by transmitting device information thereto. A leaf device may include an AI speaker or an AI camera.

In operation1410, an IoT server may identify a module required for AI calculation of data generated in a leaf device. For example, the AI calculation may include an AI vision service that analyzes video data obtained by the leaf device (e.g., IP camera) and an AI sound service that analyzes sound data obtained by the leaf device (e.g., AI speaker).

In operation1420, the IoT server may identify whether the leaf device is capable of the AI calculation. For example, the IoT server may identify whether the leaf device satisfies a hardware condition (e.g., CPU, GPU, or memory) required by the module (e.g., an AI vision module and an AI sound module) required for the AI calculation.

If a result of determination of operation1420indicates that the leaf device is capable of the AI calculation, in operation1431, the IoT server may transmit the module required for the AI calculation to the leaf device, and the leaf device may store and install the corresponding module.

In operation1433, the leaf device may use the installed module to perform on-device AI computing processing. In this case, the leaf device may transmit a result of the AI computing processing to the IoT server and AI computing need not be performed on an edge device.

If the AI calculation is unable to be performed only by the leaf device, in operation1441, an edge device and the leaf device may establish mutual connection therebetween so that the edge device dispersedly processes the AI calculation. For example, the edge device and the leaf device may broadcast, on the same local network, that each device itself is a device capable of edge computing. When the leaf device discovers the edge device, the leaf device may connect to the IoT server, identify a providable service, and then request connection. The edge device that has received a connection request may be connected to the leaf device through an authentication process.

In operation1443, the edge device may identify whether a computing required for performing the AI calculation requested by the connected at least one leaf device is sufficient. For example, if only one leaf device is connected to the edge device, the computing resource may be determined as being sufficient, and if the number of connected leaf devices is equal to or greater than a predetermined number, the computing resource is determined as being insufficient.

If the computing resource of the edge device is sufficient, in operation1445, a first service module for performing the AI calculation may be installed in the edge device, and a second service module may be installed in the leaf device.

For example, the first service module and the second service module are some of modules for performing the AI sound service, the second service module may be a voice signal pre-processing module, and the first service module may be a different recognition module. According to various embodiments, the first service module and the second service module may be changed according to a hardware characteristic (e.g., CPU or memory) of the leaf device. For example, if the leaf device does not have enough hardware resources, the second service module may include a voice signal pre-processing module, and the first service module may include a different voice recognition module. On the contrary, if the leaf device has enough hardware resources, the leaf device may include a voice recognition module including a voice signal pre-processing module.

In operation1447, the edge device and the leaf device may dispersedly perform edge AI computing by using the installed modules.

If a result of the identification of operation1443indicates that the computing required for performing the AI calculation requested by the connected at least one leaf device is insufficient, the edge device may, in operation1451, determine the priorities of the leaf devices connected to the edge device. An edge device may have more computing resources than a leaf device, but less than the cloud. Therefore, if there are many leaf devices connected to an edge device, AI distributed processing by the edge device may be difficult.

According to various embodiments, the edge device may determine the priorities between leaf devices according to sensitivity (e.g., privacy) of data transmitted from each leaf device. For example, the sensitivity of the AI vision service may be higher than that of the AI voice recognition service. In addition, the sensitivity of an AI camera for capturing an image of an indoor space may be higher than that of an AI camera for capturing an image of an outdoor space. The sensitivity and/or priority as described above may also be determined according to a user's selection. According to various embodiments, a configuration related to sensitivity of a service (e.g., AI vision service or AI voice recognition), sensitivity of a camera, and/or whether information is personal information is not limited to the described example, and may be determined (or changed) based on a user selection or designated policy information. In addition, the designated policy information may include policy information pre-stored in the leaf device1320, the edge device1310, and/or the cloud network1340.

In operation1453, the edge device may receive a service module related to a leaf device having a high priority from the IoT server, and install same. In addition, a leaf device having a low priority may be connected to a new edge device to perform AI distributed processing or may perform AI calculation through cloud computing.

An electronic device (e.g., the user device, the leaf device, and the edge device inFIG.2A) according to various embodiments disclosed herein may be various types of devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment disclosed herein is not limited to the above devices.