Patent Description:
Recently, artificial intelligence systems are being used in various fields. An artificial intelligence system is a system wherein a machine learns, determines, and becomes smarter by itself, unlike conventional rule-based smart systems. An artificial intelligence system shows a more improved recognition rate as it is used more, and becomes capable of understanding user preference more correctly. For this reason, conventional rule-based smart systems are gradually being replaced by deep learning-based artificial intelligence systems.

An artificial intelligence technology consists of machine learning (for example, deep learning) and element technologies utilizing machine learning.

Machine learning refers to an algorithm technology of classifying/learning the characteristics of input data by itself, and an element technology refers to a technology of performing functions of a human brain such as cognition and determination by using a machine learning algorithm such as deep learning, and consists of fields of technologies such as linguistic understanding, visual understanding, inference/prediction, knowledge representation, and operation control. In particular, linguistic understanding refers to a technology of recognizing languages/characters of humans, and applying/processing them, and includes natural speech processing, machine translation, communication systems, queries and answers, voice recognition/synthesis, and the like.

Recently, various electronic devices (for example, a robot cleaner, etc.) are being released while including a neural network model for object identification. Meanwhile, a neural network model stored in an electronic device cannot classify all objects due to limits in terms of performance such as the CPU or the memory capacity of the electronic device, etc., and in general, a neural network model is trained to be able to classify only objects designated by the manufacturer in advance. Accordingly, users' needs for updating a neural network model to be more appropriate for a use environment of an electronic device are increasing.

Accordingly, there is a rising need for a technology of updating a neural network model. <CIT> is directed to providing a machine learning system, a method, and an apparatus that allows a plurality of devices capable of performing machine learning to cooperate with each other to extend a coverage area of a trained model. <CIT> discloses a simulated environment to test one or more autonomous driving software stacks that include a multitude of DNNs. The publication "<NPL>et al. ) performs a study of data collection from a data management point of view.

A technical task that the disclosure aims to resolve is providing an electronic device that obtains learning data for updating a neural network model.

According to an embodiment, a method of controlling method an electronic device is provided as defined in the appended claims.

In the step of identifying the plurality of learning images, images including objects for which the probability values are smaller than a predetermined value are identified as the learning images.

The step of obtaining the learning data may include the steps of identifying at least one cluster having bigger cohesion than a predetermined value among the plurality of clusters including the feature values of the objects included in the identified learning images, and obtaining images corresponding to the feature values included in the identified at least one cluster as the learning data.

In the step of obtaining the images corresponding to the feature values included in the identified at least one cluster as the learning data, an image corresponding to a feature value that is the most approximate to the average of the plurality of feature values included in the identified at least one cluster may be obtained as the learning data.

The controlling method may further include the step of storing the obtained learning data, and in the storing step, a cluster including feature values greater than or equal to a predetermined number among the identified at least one cluster may be identified, and images corresponding to the feature values included in the identified cluster may be stored.

The learning data may be at least one among the plurality of learning images, and may include location information of pixels corresponding to the objects included in the at least one image.

The second neural network model may be a model obtained as a third neural network model having a higher performance than the first neural network model was trained based on the obtained learning data.

According to another embodiment, an electronic device is provided as defined in the appended claims.

The processor may identify images including objects for which the probability values are smaller than a predetermined value as the learning images.

The processor may identify at least one cluster having bigger cohesion than a predetermined value among the plurality of clusters including the feature values of the objects included in the identified learning images, and obtain images corresponding to the feature values included in the identified at least one cluster as the learning data.

The processor may obtain an image corresponding to a feature value that is the most approximate to the average of the plurality of feature values among the plurality of feature values included in the identified at least one cluster as the learning data.

The processor may identify a cluster including feature values greater than or equal to a predetermined number among the identified at least one cluster, and store images corresponding to the feature values included in the identified cluster.

According to an unclaimed embodiment, a neural network model update system is provided as defined in the appended claims.

According to the various embodiments of the disclosure as above, an electronic device can update a neural network model stored in the electronic device. Accordingly, the object recognition rate of the neural network model can be improved.

Other than the above, regarding effects that can be obtained or predicted from the embodiments of the disclosure, direct or implicit descriptions will be made in the detailed description for the embodiments of the disclosure. For example, regarding various effects that are predicted according to the embodiments of the disclosure, descriptions will be made in the detailed description that will be described below.

First, terms used in this specification will be described briefly, and then the disclosure will be described in detail.

As terms used in the embodiments of the disclosure, general terms that are currently used widely were selected as far as possible, in consideration of the functions described in the disclosure. However, the terms may vary depending on the intention of those skilled in the art who work in the pertinent field, previous court decisions, or emergence of new technologies, etc. Also, in particular cases, there may be terms that were designated by the applicant on his own, and in such cases, the meaning of the terms will be described in detail in the relevant descriptions in the disclosure. Accordingly, the terms used in the disclosure should be defined based on the meaning of the terms and the overall content of the disclosure, but not just based on the names of the terms.

In addition, terms such as 'the first,' 'the second,' and the like may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another component.

Further, singular expressions include plural expressions, as long as they do not obviously mean differently in the context. In addition, in the disclosure, terms such as "include" and "consist of" should be construed as designating that there are such characteristics, numbers, steps, operations, elements, components, or a combination thereof described in the specification, but not as excluding in advance the existence or possibility of adding one or more of other characteristics, numbers, steps, operations, elements, components, or a combination thereof.

Hereinafter, the embodiments of the disclosure will be described in detail with reference to the accompanying drawings, such that a person having ordinary knowledge in the technical field to which the disclosure pertains can easily carry out the disclosure. However, it should be noted that the disclosure may be implemented in various different forms, and is not limited to the embodiments described herein. Also, in the drawings, parts that are not related to explanation were omitted, for explaining the disclosure clearly, and throughout the specification, similar components were designated by similar reference numerals.

<FIG> is a diagram for illustrating a neural network model update system according to an embodiment of the disclosure. The neural network model update system may include an electronic device <NUM> and an external device <NUM>. The electronic device <NUM> is a device storing a neural network model, and it may be, for example, a robot cleaner, a smart robot, or a smart cam. The external device <NUM> is a device having a higher performance compared to the electronic device <NUM>, and it may be a device that has a faster operation processing speed, and a bigger memory capacity compared to the electronic device <NUM>. For example, the external device <NUM> may be a personal computer (PC) or a mobile terminal device. The neural network model update system may update the neural network model stored in the electronic device <NUM> by using the external device <NUM> having a relatively higher performance compared to the electronic device <NUM>. Hereinafter, a method of updating a neural network model will be described in more detail.

The electronic device <NUM> may obtain a plurality of photographed images <NUM>. Here, the electronic device <NUM> may obtain the plurality of photographed images <NUM> that photographed the surroundings of the electronic device <NUM> by using a camera installed on the electronic device <NUM>.

The electronic device <NUM> may obtain an output value <NUM> as a result of inference for an object ob by inputting the plurality of photographed images <NUM> into a first neural network model <NUM> stored in a memory <NUM>. Here, the first neural network model <NUM> is a neural network model trained to identify or recognize an object included in an image. Also, the output value <NUM> is a value for the inference result of the first neural network model <NUM> for the object ob, and it may include a classification value for the object ob and a probability value for the classification value. The classification value may mean a predicted class for the object ob. The probability value is a value indicating a probability that the object ob is an object corresponding to the classification value, i.e., a value indicating a probability that the predicted class for the object ob is the actual class of the object ob, and it may also be referred to as a score. For example, the output value <NUM> of the first neural network model <NUM> for the object ob may be "{<NUM>(a value corresponding to a cat), <NUM>(a probability that an object is a cat)}. " Also, the output value <NUM> may include information on an area wherein the object ob is located within the photographed images <NUM>. For example, the information on the area wherein the object ob is located may be coordinate information of a bounding box for the object.

Meanwhile, the first neural network model <NUM> cannot classify all objects due to limits in terms of performance such as the CPU or the memory capacity of the electronic device <NUM>, and in general, a neural network model is trained to be able to classify only objects designated by the manufacturer in advance. Accordingly, the first neural network model <NUM> needs to be updated to be more appropriate for the environment wherein the electronic device <NUM> is located. For this, the electronic device <NUM> may obtain learning data <NUM> for update of the first neural network model <NUM>.

Specifically, the electronic device <NUM> may identify an image including an object having a smaller probability value than a predetermined value (e.g., <NUM>) among the plurality of photographed images <NUM> as an image to be learned <NUM>. For example, if the probability value for the object ob obtained through the first neural network model <NUM> is smaller than the predetermined value, the electronic device <NUM> may identify the image including the object ob as the image to be learned <NUM>.

Then, the electronic device <NUM> may obtain a feature value vn for the object included in the image to be learned <NUM>. Here, the feature value is a value indicating a feature for the object, and it may also be referred to as a feature vector or feature information. The electronic device <NUM> may obtain the feature value vn for an object included in the plurality of photographed images <NUM> as an intermediate output value which is a result of inputting the plurality of photographed images <NUM> into the first neural network model <NUM>. The feature value vn is a value indicating a feature for the object ob, and it may also be referred to as a feature vector or feature information. Also, the feature value vn may correspond to the output value <NUM>. Specifically, the output value <NUM> may be obtained based on the feature value vn.

The electronic device <NUM> may perform clustering by mapping the obtained feature value vn to a random vector space. Clustering means grouping feature values having similar features by mapping feature values to a random vector space. The electronic device <NUM> may identify the clustered feature value vn in the vector space. The electronic device <NUM> may obtain the learning data <NUM> from the plurality of learning images <NUM> based on the identified feature value vn. Specifically, the electronic device <NUM> may identify a cluster cn having bigger cohesion than a predetermined value among the plurality of clusters existing in the vector space. Here, the cohesion means an index indicating how much cohered the feature information having relevance to one another is in the vector space. The electronic device <NUM> may identify a feature value that is the most approximate to the average of the plurality of feature values among the plurality of feature values included in the identified cluster cn. The electronic device <NUM> may identify the image to be learned <NUM> corresponding to the feature value included in the identified cluster as the learning data <NUM>.

Then, the electronic device <NUM> may transmit the obtained learning data <NUM> to the external device <NUM>. The external device <NUM> may obtain a second neural network model <NUM> based on the learning data <NUM>. Specifically, the external device <NUM> may train a third neural network model <NUM> having a higher performance than the first neural network model <NUM> based on the learning data <NUM>. For example, the third neural network model <NUM> may identify objects i n more various types than the first neural network model <NUM>. Then, the external device <NUM> may lighten the weight of the third neural network model <NUM>, and obtain the second neural network model <NUM>. Meanwhile, the process that the external device <NUM> obtains the second neural network model <NUM> will be described in more detail in <FIG>.

The external device <NUM> may transmit information on the obtained second neural network model <NUM> to the electronic device <NUM>. Here, the information on the second neural network model <NUM> may be information regarding the second neural network model <NUM> itself, or a parameter included in the second neural network model <NUM>. The parameter may be a weighted value. The weighted value may be a vector value including several elements.

The electronic device <NUM> may update the first neural network model <NUM> based on the information on the second neural network model <NUM>. For example, the electronic device <NUM> may store the second neural network model <NUM> in the memory <NUM>, and delete the first neural network model <NUM>. Alternatively, the electronic device <NUM> may change the parameter of the first neural network model <NUM> to the parameter of the second neural network model <NUM>.

In the above, the operation of the neural network model update system was described.

Hereinafter, the configurations of the electronic device and the external device constituting the neural network model update system will be described.

<FIG> is a block diagram illustrating a configuration of an electronic device according to an embodiment of the disclosure. Referring to <FIG>, the electronic device <NUM> may include a camera <NUM>, a communication interface <NUM>, a driving part <NUM>, a memory <NUM>, and a processor <NUM>.

The camera <NUM> may obtain a plurality of photographed images. For example, the camera <NUM> may be installed on the electronic device <NUM> and photograph the surroundings of the electronic device <NUM>, and obtain a photographed image. Meanwhile, the camera <NUM> may be implemented as cameras in various types. For example, the camera <NUM> may be implemented as an RGB camera or an IR camera based on 2D, or implemented as a Time of Flight (ToF) camera or a stereo camera based on 3D. Also, the camera <NUM> may be implemented in a form wherein two or more cameras among the aforementioned cameras are combined.

The communication interface <NUM> includes at least one circuit, and it may perform communication with external devices in various types according to communication methods in various types. The communication interface <NUM> may include a Wi-Fi chip and a Bluetooth chip. The electronic device <NUM> may perform communication with an external server or the external device <NUM> through the communication interface <NUM>.

The driving part <NUM> may be a component for moving the electronic device <NUM>. In particular, the driving part <NUM> may include an actuator for driving of the electronic device <NUM>. Also, other than the driving part <NUM>, an actuator for driving motions of other physical components (e.g., an arm, etc.) of the electronic device <NUM> may be included.

The memory <NUM> may store an operating system (OS) for controlling the overall operations of the components of the electronic device <NUM>, and instructions or data related to the components of the electronic device <NUM>. For this, the memory <NUM> may be implemented as a non-volatile memory (ex: a hard disc, a solid state drive (SSD), a flash memory), a volatile memory, etc. The memory <NUM> may store a neural network model for recognizing or identifying an object. In particular, the neural network model may be executed by a conventional generic-purpose processor (e.g., a CPU) or a separate AI-dedicated processor (e.g., a GPU, an NPU, etc.). For example, the memory <NUM> may include the first neural network model <NUM> for identifying an object. Also, the first neural network model <NUM> may be updated to the second neural network model <NUM> as described above in <FIG>.

The processor <NUM> may control the overall operations of the electronic device <NUM>. The processor <NUM> may include an object inference module <NUM>, an learning images identification module <NUM>, a learning data acquisition module <NUM>, a learning data storage module <NUM>, and a neural network model update module <NUM>. Hereinafter, operations of each module will be described in detail.

The object inference module <NUM> may obtain feature values for objects included in the plurality of photographed images <NUM> obtained through the camera <NUM>. Also, the object inference module <NUM> may obtain classification values (or, predicted classes) and probability values respectively corresponding to the feature values.

The learning images identification module <NUM> may identify learning images <NUM> among the plurality of photographed images <NUM> based on probability values for objects obtained through the object inference module <NUM>. Specifically, the learning images identification module <NUM> may identify the photographed images <NUM> including objects having probability values smaller than a predetermined value as the learning images <NUM>.

The learning data acquisition module <NUM> may obtain the learning data <NUM> based on the learning images <NUM>. Specifically, the learning data acquisition module <NUM> may cluster the feature values by mapping the feature values for the objects included in the learning images <NUM> to a random vector space. Then, the learning data acquisition module <NUM> may identify at least one cluster having bigger cohesion than a predetermined value among the plurality of clusters existing in the vector space. Also, the learning data acquisition module <NUM> may identify at least one cluster including feature values greater than or equal to a predetermined n umber among the clusters having bigger cohesion than the predetermined value.

The learning data acquisition module <NUM> may identify the learning images <NUM> corresponding to the feature values included in the identified clusters as the learning data <NUM>. Here, the learning data acquisition module <NUM> may identify a feature value that is the most approximate to the average of the plurality of feature values among the plurality of feature values included in the identified clusters. Alternatively, the learning data acquisition module <NUM> may identify a feature value that is the most approximate to the center of the identified clusters. Meanwhile, the learning data acquisition module <NUM> may identify a plurality of feature values within one cluster. Here, the learning data acquisition module <NUM> may identify a plurality of feature values within a threshold range from the average of the feature values. Alternatively, the learning data acquisition module <NUM> may identify a plurality of feature values within a threshold range from the center of the clusters.

Then, the learning data acquisition module <NUM> may identify the learning images <NUM> including the objects corresponding to the identified feature values as the learning data <NUM>. Here, the learning data <NUM> may include raw data regarding the learning images <NUM>, and location information of the objects corresponding to the identified feature information among the objects included in the learning images <NUM>. Here, the location information of the objects may be the coordinate information of the bounding boxes for the objects.

The learning data storage module <NUM> may store the obtained learning data <NUM> in the memory <NUM>. Here, the learning data storage module <NUM> may select only some of the learning data <NUM> and store it in the memory <NUM>, in consideration of the capacity of the memory <NUM>. Specifically, the learning data storage module <NUM> may identify the learning data <NUM> to be stored based on the priorities of the clusters corresponding to the obtained learning data <NUM>. Here, the priorities of the clusters may be set based on the number of the feature values included in the clusters. Also, in case a plurality of learning data <NUM> corresponding to the same cluster was obtained, the learning data storage module <NUM> may select some of the learning data <NUM> based on the feature values corresponding to the plurality of respective learning data <NUM>. For example, as a feature value is more approximate to the average value of the plurality of feature values included in the same cluster, the learning data storage module <NUM> may identify the priority of the feature value as a higher priority, and store the learning data <NUM> corresponding to the feature value based on the identified priority in the memory <NUM>. Alternatively, as a feature value is more approximate to the center of the same cluster, the learning data storage module <NUM> may identify the priority of the feature value as a higher priority, and store the learning data <NUM> corresponding to the feature value based on the identified priority in the memory <NUM>.

The neural network model update module <NUM> may update the first neural network model <NUM> based on the information on the second neural network model <NUM> received from the external device. For example, the neural network model update module <NUM> may store the second neural network model <NUM> in the memory <NUM>, and delete the first neural network model <NUM>. Alternatively, the neural network model update module <NUM> may change the parameter (or the weighted value) of the first neural network model <NUM> to the parameter of the second neural network model <NUM>.

In particular, functions related to artificial intelligence according to the disclosure are operated through the processor <NUM> and the memory <NUM>. The processor <NUM> may consist of one or a plurality of processors. Here, the one or plurality of processors may be generic-purpose processors such as a CPU, an AP, a digital signal processor (DSP), etc., graphics-dedicated processors such as a GPU and a vision processing unit (VPU), or artificial intelligence-dedicated processors such as an NPU. The one or plurality of processors may perform control such that input data is processed according to pre-defined operation rules or an artificial intelligence model stored in the memory <NUM>. Alternatively, in case the one or plurality of processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed as a hardware structure specified for processing of a specific artificial intelligence model.

The predefined operation rules or the artificial intelligence model are characterized in that they are made through learning. Here, being made through learning means that a basic artificial intelligence model is trained by using a plurality of learning data by a learning algorithm, and predefined operations rules or an artificial intelligence model set to perform desired characteristics (or, purposes) are thereby made. Such learning may be performed in a device itself wherein artificial intelligence is performed according to the disclosure, or through a separate server and/or system. As examples of learning algorithms, there are supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

An artificial intelligence model may be made through learning. Here, being made through learning means that a basic artificial intelligence model is trained by using a plurality of learning data by a learning algorithm, and predefined operations rules or an artificial intelligence model set to perform desired characteristics (or, purposes) are thereby made. An artificial intelligence model may consist of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and performs a neural network operation through the operation result of the previous layer and an operation among <NUM> the plurality of weight values. The plurality of weight values included by the plurality of neural network layers may be optimized by the learning result of the artificial intelligence model. For example, the plurality of weight values may be updated such that a loss value or a cost value obtained at the artificial intelligence model during a learning process is reduced or minimized.

Visual understanding is a technology of recognizing an object in a similar manner to human vision, and processing the object, and includes object recognition, object tracking, image retrieval, human recognition, scene recognition, space recognition (3D reconstruction/localization), image enhancement, etc..

An artificial neural network may include a deep neural network (DNN), and there are, for example, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a generative adversarial network (GAN), a restricted Boltzmann Machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks, etc.

<FIG> is a diagram illustrating a configuration of an external device according to an embodiment of the disclosure. Referring to <FIG>, the external device <NUM> may include a communication interface <NUM>, a memory <NUM>, and a processor <NUM>. Meanwhile, the communication interface <NUM> may correspond to the communication interface <NUM> in <FIG>, and thus detailed explanation in that regard will be omitted.

The memory <NUM> may store the third neural network model <NUM> and the fourth neural network model <NUM>. Here, the third neural network model <NUM> may be the original model of the first neural network model <NUM> in <FIG>. That is, the first neural network model <NUM> may be a model which is a weight-lighted form of the third neural network model <NUM>. Also, the fourth neural network model <NUM> may be a model having a higher performance than the third neural network model <NUM>, and it may be a model trained to be able to identify a larger number of objects than the third neural network model <NUM>. Also, the memory <NUM> may store an operating system (OS) for controlling the overall operations of the components of the external device <NUM>, and instructions or data related to the components of the external device <NUM>. For this, the memory <NUM> may be implemented as a non-volatile memory (ex: a hard disc, a solid state drive (SSD), a flash memory), a volatile memory, etc..

The processor <NUM> may control the overall operations of the external device <NUM>. The processor <NUM> may include a learning data inference module <NUM>, a neural network model training module <NUM>, and a neural network model weight lightening module <NUM>. Hereinafter, the operations of each module will be described in detail.

The learning data inference module <NUM> may obtain classification values and probability values for objects included in the learning data <NUM> by inputting the learning data <NUM> obtained from the electronic device <NUM> into the fourth neural network model <NUM>. As the fourth neural network model <NUM> has a higher performance than the first neural network model <NUM>, the learning data inference module <NUM> may obtain classification values and probability values for objects that cannot be identified by the first neural network model <NUM> by using the fourth neural network model <NUM>. Then, the learning data inference module <NUM> may identify an object having a bigger probability value than a predetermined probability value (e.g., <NUM>), and perform a labeling operation for the learning data <NUM> including the identified object. That is, the learning data inference module <NUM> may match the learning data <NUM> and classification values for objects included in the learning data <NUM>, and store them in the memory <NUM>. Hereinafter, the learning data <NUM> labeled by the learning data inference module <NUM> will be referred to as the labelled data <NUM>.

The neural network model training module <NUM> may train the third neural network model <NUM> based on the labelled data <NUM>. Here, the neural network model training module <NUM> may train the third neural network model <NUM> based on the labelled data <NUM> and the learning data for the first neural network model <NUM>. Accordingly, a phenomenon that the third neural network model <NUM> is biased or overfitted can be prevented. Meanwhile, the neural network model training module <NUM> trains the third neural network model <NUM> which is the original model (i.e., the model before weight lightening) of the first neural network model <NUM>, but not the first neural network model <NUM>, and accordingly, the learning time can be shortened.

The neural network model weight lightening module <NUM> may obtain the second neural network model <NUM> by lightening the weight of the trained third neural network model <NUM>. Specifically, the neural network model weight lightening module <NUM> may obtain the second neural network model <NUM> by converting the data type of the weighted value of the third neural network model <NUM>. For example, in case the weighted value of the third neural network model <NUM> is a value of a 32bit float type, the neural network model weight lightening module <NUM> may convert the weighted value of the third neural network model <NUM> to a value of an 8bit integer type. That is, the neural network model weight lightening module <NUM> may obtain the second neural network model <NUM> by reducing the size of the weighted value of the third neural network model <NUM>. Like this, the neural network model weight lightening module <NUM> may obtain the second neural network model <NUM> which is a weight-lighted form of the third neural network model <NUM>. Then, the processor <NUM> may transmit the second neural network model <NUM> to the electronic device <NUM> through the communication interface <NUM>. Accordingly, even in case the capacity of the memory <NUM> of the electronic device <NUM> is restrictive, the electronic device <NUM> may update the first neural network model <NUM> to the second neural network model <NUM>.

<FIG> is a diagram for illustrating a method of identifying learning images according to an embodiment of the disclosure.

The electronic device <NUM> may obtain classification values and probability values for objects included in the plurality of respective photographed images <NUM> by respectively inputting the plurality of photographed images <NUM> into the first neural network model <NUM>. For example, the electronic device <NUM> may obtain a classification value ('clothes') and a probability value ('<NUM>') for a first object ob1 by inputting a first photographed image <NUM>-<NUM> into the first neural network model <NUM>. The electronic device <NUM> may obtain a classification value ('an electric wire') and a probability value ('<NUM>') for a second object ob2 by inputting a second photographed image <NUM>-<NUM> into the first neural network model <NUM>. The electronic device <NUM> may obtain a classification value ('vinyl') and a probability value ('<NUM>') for a third object ob3 by inputting a third photographed image <NUM>-<NUM> into the first neural network model <NUM>. The electronic device <NUM> may obtain a classification value ('clothes') and a probability value ('<NUM>') for a fourth object ob4 by inputting a fourth photographed image <NUM>-<NUM> into the first neural network model <NUM>. Meanwhile, in <FIG>, a classification value for each object was expressed in a word indicating the object, for the convenience of explanation, but as described above, a classification value may be a numerical value corresponding to each object.

Then, the electronic device <NUM> may identify the learning images <NUM> based on the probability values for the objects included in the plurality of respective photographed images <NUM>. Specifically, the electronic device <NUM> may identify images including objects for which the probability values are smaller than a predetermined value among the plurality of photographed images <NUM> as the learning images <NUM>. Here, the predetermined value may mean a value for evaluating the recognition rate or the learning degree of the electronic device <NUM> for objects. The electronic device <NUM> needs to be additionally trained for objects having smaller probability values than the predetermined value. For example, in case the predetermined value is <NUM>, the electronic device <NUM> may identify the first photographed image <NUM>-<NUM>, the second photographed image <NUM>-<NUM>, and the third photographed image <NUM>-<NUM> respectively including the first object ob1, the second object ob2, and the third object ob3 for which the probability values are smaller than <NUM> as the learning images <NUM>. In contrast, the electronic device <NUM> may not identify the fourth photographed image <NUM>-<NUM> including the fourth object ob4 for which the probability value is bigger than <NUM> as the image to be learned <NUM>.

<FIG> and <FIG> are diagrams for illustrating a method of obtaining learning data of an electronic device according to an embodiment of the disclosure.

Referring to <FIG>, the electronic device <NUM> may cluster the feature values by mapping the feature values of the plurality of respective learning images <NUM> to a vector space. Specifically, the electronic device <NUM> may cluster the feature values by mapping the first to third feature values v<NUM>, v<NUM>, v<NUM> respectively corresponding to the first object to the third object ob1, ob2, ob3 to a vector space. Accordingly, in the vector space, a plurality of clusters c<NUM>, c<NUM>, c<NUM> including at least one feature value may be formed. Here, the first to third feature values v<NUM>, v<NUM>, v<NUM> may be obtained by using the first neural network model <NUM>, as described above. Meanwhile, in <FIG>, the vector space was illustrated as a three-dimensional space, but this is merely an example, and the vector space may be a space in a high dimension greater than or equal to a three dimension. Also, each axis x, y, z of the vector space may have various factors, and may have, for example, factors such as colors, sizes, texture, shapes, etc..

Referring to <FIG>, the electronic device <NUM> may identify clusters having bigger cohesion than a predetermined value among the plurality of clusters c<NUM>, c<NUM>, c<NUM> existing in the vector space. For example, the electronic device <NUM> may identify the first cluster c<NUM> and the second cluster c<NUM> among the plurality of clusters c<NUM>, c<NUM>, c<NUM>.

The electronic device <NUM> may obtain photographed images corresponding to the feature values included in the first cluster c<NUM> and the second cluster c<NUM> as the learning data <NUM>. Here, the electronic device <NUM> may identify a feature value that is the most approximate to the average value of the feature values included in the first cluster c<NUM>. For example, the electronic device <NUM> may identify the <NUM>-<NUM> feature value v<NUM>-<NUM> included in the first cluster c<NUM>. Then, the electronic device <NUM> may obtain an image to be learned including an object corresponding to the <NUM>-<NUM> feature value v<NUM>-<NUM> as the first learning data <NUM>-<NUM>. Here, the first learning data <NUM>-<NUM> may include location information regarding the area R1 of the object in the <NUM>-<NUM> feature value v<NUM>-<NUM>. Likewise, the electronic device <NUM> may identify the <NUM>-<NUM> feature value v<NUM>-<NUM> that is the most approximate to the average value of the feature values included in the second cluster c<NUM>, and obtain an image to be learned including an object corresponding to the <NUM>-<NUM> feature value v<NUM>-<NUM> as the second learning data <NUM>-<NUM>.

Meanwhile, the electronic device <NUM> may identify a plurality of feature values for each cluster, and obtain a plurality of learning images corresponding to the plurality of respective identified feature values as the learning data. Here, the electronic device <NUM> may identify feature values that exist within a predetermined distance from the center of each cluster, and obtain a plurality of learning images including objects corresponding to the identified feature values as the learning data.

<FIG> is a diagram for illustrating a method of identifying learning images according to another embodiment of the disclosure. Specifically, <FIG> is a diagram for illustrating a method of identifying learning images in case a plurality of objects are included in the photographed images <NUM>.

In case a plurality of objects are included in the photographed images <NUM>, if even one object having a smaller probability value than the predetermined value exists among the plurality of objects, the electronic device <NUM> may identify the photographed images <NUM> as the learning images <NUM>. Referring to <FIG>, the photographed images <NUM> may include a fifth object ob5, a sixth object ob6, and a seventh object ob7. Also, the probability value of the fifth object ob5 may be <NUM>, the probability value of the sixth object 0b6 may be <NUM>, and the probability value of the seventh object ob7 may be <NUM>. In case the predetermined probability value is <NUM>, as the probability value of the fifth object ob5 is smaller than the predetermined value, the electronic device <NUM> may identify the photographed images <NUM> as the learning images <NUM>.

<FIG> is a diagram for illustrating a method of clustering according to another embodiment of the disclosure.

The electronic device <NUM> may cluster the feature values by mapping feature values for objects included in the learning images to a vector space. Here, the electronic device <NUM> may identify feature values for objects having smaller probability values than the predetermined value, and map the identified feature values to a vector space. Referring to <FIG>, the electronic device <NUM> may identify the fifth feature value v<NUM> for the fifth object ob5 having a smaller probability value than the predetermined value, and cluster the feature value by mapping the fifth feature value v<NUM> to the vector space. In contrast, the electronic device <NUM> may not map the sixth feature value v<NUM> and the seventh feature value v<NUM> for the sixth object ob6 and the seventh object ob7 having bigger probability values than the predetermined value to the vector space.

As described above, the electronic device <NUM> may obtain the learning data <NUM> and transmit it to the external device <NUM>. Then, the external device <NUM> may obtain the second neural network model <NUM> based on the learning data <NUM>. Hereinafter, the method for the external device <NUM> to obtain the second neural network model <NUM> will be described.

<FIG> is a diagram for illustrating a method of obtaining labelled data according to an embodiment of the disclosure.

The external device <NUM> may obtain the learning data <NUM> including the first learning data <NUM>-<NUM> and the second learning data <NUM>-<NUM> from the electronic device <NUM>. The external device <NUM> may obtain classification values and probability values for objects included in the learning data <NUM> by inputting the learning data <NUM> into the fourth neural network model <NUM>. For example, the external device <NUM> may obtain a classification value ('clothes') and a probability value ('<NUM>') for the first object ob1 by inputting the first learning data <NUM>-<NUM> into the fourth neural network model <NUM>, and. Also, the external device <NUM> may obtain a classification value ('an electric wire') and a probability value ('<NUM>') for the second object ob2 by inputting the second learning data <NUM>-<NUM> into the fourth neural network model <NUM>. As described above, the fourth neural network model <NUM> is a model having a higher performance compared to the first neural network model <NUM>, and it may include a larger number of layers than the first neural network model <NUM>.

The external device <NUM> may identify an object having a bigger probability value than the predetermined value. For example, in case the predetermined value is <NUM>, the external device <NUM> may identify the first object ob1 for which the probability value is bigger than <NUM>. Then, the external device <NUM> may perform a labeling operation for the identified object. Here, the external device <NUM> may match the first learning data <NUM>-<NUM> including the first object ob1 and the classification value ('clothes') for the first object ob1, and store them. Accordingly, the external device <NUM> may obtain the labelled data <NUM>.

Referring to <FIG>, the external device <NUM> may train the third neural network model <NUM> based on the labelled data <NUM>. The third neural network model <NUM> may be trained to be able to identify objects included in the labelled data <NUM>. Here, the external device <NUM> may train the third neural network model <NUM> based on supervised learning.

Meanwhile, the external device <NUM> may train the third neural network model <NUM> based on the labelled data <NUM> and the learning data <NUM>. Here, the learning data <NUM> is data used for training of the first neural network model <NUM>, and it may be stored in advance in the external device <NUM>. Accordingly, a phenomenon that the third neural network model <NUM> is biased or overfitted with respect to the labelled data <NUM> can be prevented.

The external device <NUM> may obtain the second neural network model <NUM> by lightening the weight of the third neural network model <NUM>. Specifically, the external device <NUM> may obtain the weighted value of the second neural network model <NUM> by reducing the size of the data type of the weighted value (or the parameter) of the third neural network model <NUM>. Referring to <FIG>, the third neural network model <NUM> may have the <NUM>-<NUM> weighted value w1-<NUM> and the <NUM>-<NUM> weighted value w2-<NUM> of the third data type <NUM>. For example, the third data type <NUM> may be a 32bit float type. Also, the second neural network model <NUM> may have the <NUM>-<NUM> weighted value w1-<NUM> and the <NUM>-<NUM> weighted value w2-<NUM> of the second data type <NUM>. For example, the second data type <NUM> may be an 8bit integer type.

As illustrated in <FIG>, as the external device <NUM> converts the third data type <NUM> to the second data type <NUM>, the <NUM>-<NUM> weighted value w1-<NUM> and the <NUM>-<NUM> weighted value w2-<NUM> may be obtained from the <NUM>-<NUM> weighted value w1-<NUM> and the <NUM>-<NUM> weighted value w2-<NUM>. Here, the external device <NUM> may obtain values that are respectively the most approximate to the <NUM>-<NUM> weighted value w1-<NUM> and the <NUM>-<NUM> weighted value w2-<NUM> as the <NUM>-<NUM> weighted value w1-<NUM> and the <NUM>-<NUM> weighted value w2-<NUM>. For example, the external device <NUM> may obtain the value which is the most approximate to the <NUM>-<NUM> weighted value w1-<NUM> among the values of the second data type <NUM> as the <NUM>-<NUM> weighted value w1-<NUM>. Here, the <NUM>-<NUM> weighted value w1-<NUM> and the <NUM>-<NUM> weighted value w1-<NUM> may have the same value. Also, the external device <NUM> may obtain the <NUM>-<NUM> weighted value w2-<NUM> which is the most approximate value to the <NUM>-<NUM> weighted value w2-<NUM> among the values of the second data type <NUM>.

The external device <NUM> may transmit information on the obtained second neural network model <NUM> to the electronic device <NUM>. Specifically, the external device <NUM> may transmit the second neural network model <NUM> to the electronic device <NUM>. Alternatively, the external device <NUM> may transmit the parameter (or the weighted value) of the second neural network model <NUM> to the electronic device <NUM>. Meanwhile, the external device <NUM> may transmit the second neural network model <NUM> or the parameter (or the weighted value) of the second neural network model <NUM> to the electronic device <NUM> based on the available capacity of the memory <NUM>. For example, if the available capacity of the memory <NUM> is smaller than a predetermined value, the external device <NUM> may transmit only parameter information excluding the configuration information for the second neural network model <NUM> to the electronic device <NUM>. Alternatively, if the available capacity of the memory <NUM> is bigger than the predetermined value, the external device <NUM> may transmit the second neural network model <NUM> itself including the configuration information and the parameter information for the second neural network model <NUM>.

The electronic device <NUM> may update the first neural network model <NUM> based on the received information on the second neural network model <NUM>. For example, if the second neural network model <NUM> was received, the electronic device <NUM> may store the second neural network model <NUM> in the memory <NUM>, and delete the pre-stored first neural network model <NUM>. Alternatively, in case the parameter of the second neural network model <NUM> was received, the electronic device <NUM> may update the first neural network model <NUM> by changing the parameter of the first neural network model <NUM> to the parameter of the second neural network model <NUM>.

In the above, the operations of the electronic device and the external device were described.

Hereinafter, a controlling method of the electronic device and the external device will be described.

<FIG> is a flow chart illustrating a controlling method of an electronic device according to an embodiment of the disclosure. Referring to <FIG>, the electronic device <NUM> may obtain a plurality of photographed images in operation S710. Here, the electronic device <NUM> may obtain photographed images that photographed the surroundings of the electronic device <NUM> through the camera <NUM>. Alternatively, the electronic device <NUM> may obtain a plurality of photographed images from the external device through the communication interface <NUM>.

The electronic device <NUM> may obtain feature values for objects included in the plurality of photographed images by inputting the plurality of photographed images into the first neural network model, classification values for classifying the objects, and probability values for the classification values in operation S720. Here, the classification values (or the predicted classes) and the probability values may be obtained as the feature values for the objects are input into an output layer included in the first neural network model <NUM>. Accordingly, the classification values and the probability values may respectively correspond to the feature values.

The electronic device <NUM> may identify a plurality of learning images among the plurality of photographed images based on the obtained probability values in operation S730. Here, the electronic device <NUM> may identify photographed images including objects having smaller probability values than a predetermined value as the learning images.

The electronic device <NUM> may identify the clustered feature values in operation S740 by mapping the feature values of the objects included in the plurality of respective identified learning images to a vector space. Here, the electronic device <NUM> may identify clusters having bigger cohesion than a predetermined value among the plurality of clusters existing in the vector space, and identify feature values included in the identified clusters.

The electronic device <NUM> may obtain learning data from the plurality of learning images based on the identified feature values in operation S750. Here, the electronic device <NUM> may identify a feature value that is the most approximate to the average value of the plurality of feature values included in the identified clusters. Then, the electronic device <NUM> may obtain an image to be learned including an object corresponding to the identified feature value as the learning data.

The electronic device <NUM> may transmit the obtained learning data to the external device in operation S760. Here, the learning data may include raw data of the image to be learned (or the photographed image) and location information of the object within the image to be learned.

The electronic device <NUM> may receive information on the second neural network model from the external device in operation S770. Then, the electronic device <NUM> may update the first neural network model based on the information on the second neural network model in operation S780. Here, in case the second neural network model <NUM> was received, the electronic device <NUM> may store the second neural network model <NUM> in the memory <NUM>, and delete the pre-stored first neural network model <NUM>. Alternatively, in case the parameter of the second neural network model <NUM> was received, the electronic device <NUM> may update the first neural network model <NUM> by changing the parameter of the first neural network model <NUM> to the parameter of the second neural network model <NUM>.

<FIG> is a flow chart illustrating a controlling method of an electronic device according to another embodiment of the disclosure. Referring to <FIG>, the electronic device <NUM> may identify a plurality of learning images in operation S730, and store the identified learning images in operation S735. Here, the electronic device <NUM> may store the plurality of identified learning images in the memory.

Then, the electronic device <NUM> may identify whether an idle resource exists in operation S737. Specifically, the electronic device <NUM> may identify whether the idle value of the processor <NUM> is greater than or equal to a predetermined value (e.g., <NUM>%). Alternatively, the electronic device <NUM> may identify whether the use amount of the memory <NUM> is smaller than or equal to a predetermined value (e.g., <NUM>%).

If it is identified that an idle resource does not exist, the electronic device <NUM> may return to the operation S710 in <FIG>, and obtain a plurality of photographed images. Then, the electronic device <NUM> may identify a plurality of learning images among the plurality of obtained photographed images in operation S730, and store the images in operation S735.

If it is identified that an idle resource exists, the electronic <NUM> may identify clustered feature values by mapping feature values of objects included in the plurality of respective learning images to a vector space as in the operation S740 in <FIG>, and. Then, the electronic device <NUM> may obtain learning data from the plurality of learning images based on the identified feature values in operation S750. Then, the electronic device <NUM> may store the obtained learning data in operation S755.

The electronic device <NUM> may identify whether the electronic device <NUM> is in a standby mode in operation S757. Here, the standby mode may be a mode wherein the electronic device <NUM> is charged by an external power device. If the electronic device <NUM> is not identified to be in the standby mode, the electronic device <NUM> may return to the operation S710 in <FIG>, and obtain a plurality of photographed images. Then, the electronic device <NUM> may identify a plurality of learning images among the plurality of obtained photographed images in operation S730, and store the images in operation S735.

If the electronic device <NUM> is identified to be in the standby mode, the electronic device <NUM> may transmit the obtained learning data to the external device as in the operation S760 in <FIG>. Here, the obtained learning data is the learning data stored in the operation S755. Then, the electronic device <NUM> may receive information on the second neural network model from the external device in operation S770, and update the first neural network model based on the information on the second neural network model in operation S780. Meanwhile, in the above, it was described that, when the electronic device <NUM> enters the standby mode, the electronic device <NUM> transmits the learning data to the external device. However, this is merely an example, and the electronic device <NUM> may transmit the stored learning data to the external device <NUM> as a user instruction for updating the first neural network model <NUM> is input.

<FIG> is a sequence diagram for illustrating a neural network model update system. The electronic device <NUM> may obtain a plurality of photographed images in operation S810. As the operation corresponds to the operation S710, detailed explanation in this regard will be omitted.

The electronic device <NUM> may obtain learning data from the plurality of photographed images by using the first neural network model in operation S820. Specifically, the electronic device <NUM> may obtain learning data through the operations S720, S730, S740, and S750 in FIG. Then, the electronic device <NUM> may transmit the learning data to the external device <NUM> in operation S830. The external device <NUM> may obtain information on the second neural network model based on the learning data in operation S840. Here, the external device <NUM> may obtain labelled data <NUM> from the learning data <NUM> by using the fourth neural network model <NUM>. Then, the external device <NUM> may train the third neural network model <NUM> based on the labelled data <NUM>, and lighten the weight of the third neural network model <NUM>, and obtain information on the second neural network model <NUM>.

Then, the electronic device <NUM> may receive the information on the second neural network model from the external device <NUM> in operation S850, and update the first neural network model based on the information on the second neural network model in operation S860.

Meanwhile, the various embodiments described above may be implemented in a recording medium that can be read by a computer or a device similar to a computer, by using software, hardware, or a combination thereof. In some cases, the embodiments described in this specification may be implemented as a processor itself. According to implementation by software, the embodiments such as procedures and functions described in this specification may be implemented as separate software modules. Each of the software modules can perform one or more functions and operations described in this specification.

Meanwhile, computer instructions for performing processing operations according to the aforementioned various embodiments of the disclosure may be stored in a non-transitory computer-readable medium. Computer instructions stored in such a non-transitory computer-readable medium may make the processing operations according to the aforementioned various embodiments performed by a specific machine, when the instructions are executed by the processor of the specific machine.

A non-transitory computer-readable medium refers to a medium that stores data semi-permanently, and is readable by machines, but not a medium that stores data for a short moment such as a register, a cache, and a memory. As specific examples of a non-transitory computer-readable medium, there may be a CD, a DVD, a hard disc, a blue-ray disc, a USB, a memory card, a ROM and the like.

Meanwhile, a storage medium that is readable by machines may be provided in the form of a non-transitory storage medium. Here, the term 'a non-transitory storage medium' only means that the device is a tangible device, and does not include a signal (e.g.: an electronic wave), and the term does not distinguish a case wherein data is stored semi-permanently in a storage medium and a case wherein data is stored temporarily. For example, 'a non-transitory storage medium' may include a buffer wherein data is temporarily stored.

Also, according to an embodiment, methods according to the various embodiments disclosed herein may be provided while being included in a computer program product. A computer program product refers to a product, and it can be traded between a seller and a buyer. A computer program product can be distributed in the form of a storage medium that is readable by machines (e.g.: a compact disc read only memory (CD-ROM)), or distributed directly on-line (e.g.: download or upload) through an application store (e.g.: Play Store™), or between two user devices (e.g.: smartphones). In the case of on-line distribution, at least a portion of a computer program product (e.g.: a downloadable app) may be stored in a storage medium readable by machines such as the server of the manufacturer, the server of the application store, and the memory of the relay server at least temporarily, or may be generated temporarily.

Claim 1:
A method of controlling an electronic device (<NUM>), the method comprising:
obtaining (S710) a plurality of images, wherein the plurality of images include one or more objects;
inputting (S720) the plurality of images into a first neural network model (<NUM>) for identifying objects included in the inputted images, and based on the inputting of the plurality of images into the first neural network model (<NUM>) for identifying objects, obtaining a feature value for each object of the one or more objects, a predicted class for each object of the one or more objects based on the respective obtained feature value, and a probability value for the predicted class for each of the one or more objects;
identifying (S730) one or more learning images including objects from among the one or more objects for which the obtained probability value is less than a predetermined value;
identifying (S740) one or more clusters of feature values by mapping the feature values of the one or more objects included in the one or more identified learning images to a vector space;
obtaining (S750) learning data from the one or more identified learning images based on the obtained feature values;
transmitting (S760) the obtained learning data to an external device (<NUM>);
receiving (S770) information on a second neural network model (<NUM>) from the external device (<NUM>); and
updating (S780) the first neural network model (<NUM>) based on the received information on the second neural network model (<NUM>).