Patent ID: 12229965

Throughout the drawings and the detailed description, unless otherwise described or provided, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The drawings may not be to scale, and the relative size, proportions, and depiction of elements in the drawings may be exaggerated for clarity, illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatuses, and/or systems described herein will be apparent after an understanding of the disclosure of this application. For example, the sequences of operations described herein are merely examples, and are not limited to those set forth herein, but may be changed as will be apparent after an understanding of the disclosure of this application, with the exception of operations necessarily occurring in a certain order.

The features described herein may be embodied in different forms and are not to be construed as being limited to the examples described herein. Rather, the examples described herein have been provided merely to illustrate some of the many possible ways of implementing the methods, apparatuses, and/or systems described herein that will be apparent after an understanding of the disclosure of this application.

The terminology used herein is for describing various examples only and is not to be used to limit the disclosure. The articles “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms “comprises,” “includes,” and “has” specify the presence of stated features, numbers, operations, members, elements, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, members, elements, and/or combinations thereof.

Throughout the specification, when a component is described as being “connected to,” or “coupled to” another component, it may be directly “connected to,” or “coupled to” the other component, or there may be one or more other components intervening therebetween. In contrast, when an element is described as being “directly connected to,” or “directly coupled to” another element, there can be no other elements intervening therebetween.

Although terms such as “first,” “second,” and “third” may be used herein to describe various members, components, regions, layers, or sections, these members, components, regions, layers, or sections are not to be limited by these terms. Rather, these terms are only used to distinguish one member, component, region, layer, or section from another member, component, region, layer, or section. Thus, a first member, component, region, layer, or section referred to in the examples described herein may also be referred to as a second member, component, region, layer, or section without departing from the teachings of the examples.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and based on an understanding of the disclosure of the present application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of the present application and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Also, in the description of example embodiments, detailed description of structures or functions that are thereby known after an understanding of the disclosure of the present application will be omitted when it is deemed that such description will cause ambiguous interpretation of the example embodiments. Hereinafter, examples will be described in detail with reference to the accompanying drawings, and like reference numerals in the drawings refer to like elements throughout.

FIG.1illustrates an example of image segmentation of an electronic device.

Referring toFIG.1, an electronic device100of an example may generate image segmentation data120in which an input image is segmented by class and/or cluster. In the example ofFIG.1, the electronic device100may be provided in a vehicle. The electronic device100integrated into the vehicle may include, as an image acquirer, an imaging module (e.g., a camera sensor) that captures a scene110around (e.g., in front of) the vehicle. However, the image acquirer is not limited to the imaging module, and may be a communicator that receives an image from an external imaging device.

The electronic device100may generate an input image corresponding to the scene110by capturing an image of the scene110through a camera sensor. For example, the image acquirer of the electronic device100may obtain the image of the scene110around the vehicle while the vehicle is traveling.

For example, a processor of the electronic device100provided in the vehicle may generate information associated with the traveling of the vehicle using the image segmentation data120. The information associated with the traveling of the vehicle may be data used to assist in the traveling of the vehicle or used for the traveling of the vehicle, and include path guidance information, danger warning information (e.g., information on an accident such as a collision), road condition information (e.g., road congestion), and surrounding environment information (e.g., a distance map indicating distances to nearby objects). For example, the electronic device100may identify an object from the image segmentation data120, calculate a distance to the identified object, and generate a distance map indicating the calculated distance to the identified object. However, examples of the information associated with the traveling are not limited thereto. The electronic device100may visually display the information associated with the traveling through a display. In the example ofFIG.1, the electronic device100may output the image segmentation data120to a display panel of the vehicle.

The electronic device100provided in the vehicle may perform autonomous driving using the image segmentation data120. The processor of the electronic device100may perform one of or a combination of two or more of speed control, acceleration control, and steering control using the image segmentation data120while the vehicle is traveling. For example, the processor may calculate a distance to an object present nearby based on the image segmentation data120, and perform one of or a combination of two or more of changing (e.g., increasing or decreasing) the speed of the vehicle, changing (e.g., increasing or decreasing) the acceleration of the vehicle, and changing the steering of the vehicle, based on the calculated distance to the object.

To generate the image segmentation data120, the electronic device100may calculate (or obtain) a class map and a cluster map from the input image based on an image segmentation model. The class map may be a map representing a class labeled in the input image, and may be a set of elements indicating classes labeled in pixels of the input image. For example, a value of an element of a class map corresponding to one class may indicate a probability that a pixel corresponding to the element belongs to the class. The electronic device100may also calculate (or obtain) a class map for each of a plurality of classes. The cluster map may be a map representing a clustering result from the input image, and may be a set of elements indicating clusters mapped for pixels of the input image. For example, a value of an element of a cluster map corresponding to one cluster may indicate a probability that a pixel corresponding to the element belongs to the cluster. The class map and cluster map will be described later in detail with reference toFIG.3.

The electronic device100may label a class for at least a portion or region in the input image. For example, the electronic device100may label a class for each pixel of the input image. The electronic device100may label a background class for a pixel that does not correspond to a foreground or an object. The electronic device100may apply a clustering result to a pixel for which a foreground class or an object class is not labeled, for example, a pixel for which the background class is labeled. As will be described later, the electronic device100may train a classifier layer that outputs the class map and a clustering layer that outputs the cluster map together. The electronic device100may train the image segmentation model such that it learns both class labeling and clustering. Thus, while classifying the input image into a limited number of classes, the electronic device100may perform clustering on a portion in the input image in which an object being out of an available class is shown. The electronic device100may dynamically generate the image segmentation data120even when the available classes are limited. Even when the number of object types (e.g., people, things, vehicles, animals, signs, and roads) for which recognition is required in one scene (e.g., the scene110) during the traveling of the vehicle exceeds the number of available classes, the electronic device100may provide a flexible image segmentation result through the clustering layer.

The electronic device100may upgrade the existing image segmentation model when another class is to be added in addition to a currently available class. For example, the electronic device100may upgrade the image segmentation model by replacing, in the image segmentation model, the existing classifier layer with a new classifier layer having an increased number of available classes. The upgrading of the image segmentation model will be described later in detail with reference toFIG.6.

Although the example in which the electronic device100using the image segmentation model is provided in the vehicle is mainly described herein, examples are not limited thereto. For another example, the electronic device100may be provided in an augmented reality (AR) providing device. In this example, the processor of the electronic device100may generate an AR content using the image segmentation data120. The AR content may be a content provided to a user in AR and may include, for example, a content mapped to a nearby object and/or background. The electronic device100may calculate a distance and/or position to and/or of an identified object and/or background based on the user by using the image segmentation data120, and generate the AR content using the calculated distance and/or position. For example, the electronic device100may generate the AR content (e.g., a path guidance content such as an arrow indicating a direction) having the position corresponding to the identified object (e.g., a road). The electronic device100may visually provide the user with the generated AR content through an AR display. The display may include a head-up display (HUD), for example. For example, the electronic device100may visually output the AR content to the position determined as described above.

As described above, the electronic device100may generate the image segmentation data120of objects shown in the scene110, even though labels are not formed for all the objects shown in the scene110of the input image. In addition, even after the image segmentation model is provided, the electronic device100may perform continual learning based on a previously learned knowledge for the image segmentation model and may thus improve the cost and time efficiency of learning. Further, in continual learning, the electronic device100may prevent a collision between a previously learned knowledge (e.g., an existing available class) on a portion without an object class (e.g., a background class) and a new knowledge to be learned (e.g., a class to be newly added). Continual learning will be described later in detail with reference toFIG.6.

FIG.2illustrates an example of an image segmentation method.FIG.3illustrates an example of an image segmentation network.

An electronic device of an example may generate a class map325and a cluster map335from an input image301using an image segmentation model300. The image segmentation model300may be a machine learning model designed and trained to output a class map and a cluster map from an image, and may include a neural network.

The neural network may include, for example, a deep neural network (DNN). The DNN may include a fully connected network (FCN), a deep convolutional network (DCN), and a recurrent neural network (RNN). The neural network may perform object classification, object recognition, and image recognition by mapping input data and output data that are in a nonlinear relationship with each other through deep learning, and is described herein as performing mainly image segmentation. Image segmentation may be an operation of identifying a portion and/or region in an image that represents the same element (e.g., an object or background), for example, an operation (e.g., class labeling) of labeling pixels representing the same element as the same class and an operation of clustering pixels representing the same element into the same cluster. For reference, the clustering may be construed as an operation of pseudo-labeling, as an undefined class, pixels corresponding to portions and/or regions identified as representing the same element.

Deep learning may be a machine learning method used to solve a problem such as image segmentation from a big dataset, and may map input data and output data to each other through supervised or unsupervised learning. Although an example in which the neural network is trained mainly through supervised learning is described herein, examples are not limited thereto. The training will be described later in detail with reference toFIGS.5and6.

The neural network of the image segmentation model300may be a model of a machine learning structure designed to extract feature data from input data and provide an inference operation using the extracted feature data. The feature data may be data associated with a feature abstracted from the input data (e.g., the input image301). For example, when the input data is an image, the feature data may be data abstracted from the image and may be represented in the form of a vector, for example. In an example, the inference operation using the feature data may include class labeling and clustering. Class labeling may be an operation of labeling each pixel of the input image301into a class. Clustering may be an operation of clustering pixels representing the same or similar object among pixels of the input image301into the same cluster.

For example, the image segmentation model300illustrated inFIG.3may include a neural network, and the neural network may include a feature extraction layer310, a classifier layer320, and a clustering layer330. As described above, the feature extraction layer310may be a layer for extracting feature data from input data and include, for example, a convolution layer for a convolution operation. The convolution layer may be a layer in which a kernel filter-based convolution operation is applied to data. The classifier layer320may be a layer that outputs the class map325from the feature data. The clustering layer330may be a layer that outputs the cluster map335from the feature data. A layer configuration of the feature extraction layer310, the classifier layer320, and the clustering layer330is not limited to a particular example, but may include various combinations of various layers including, for example, a convolutional layer, a pooling layer, a dense layer, and a fully connected layer.

As illustrated inFIG.3, the classifier layer320and the clustering layer330may be connected as layers subsequent to the feature extraction layer310. That is, the classifier layer320and the clustering layer330may receive extracted common feature data from the feature extraction layer310. The classifier layer320and the clustering layer330may each include one or more layers, and each layer may include one or more nodes.

For reference, nodes of a layer may be connected to nodes of a subsequent layer of the layer through links. The number of the links may correspond to the number of the nodes included in the subsequent layer, and a weight may be set for each of the links. A link may also be referred to as a connecting line. To each node included in a layer, an output of an activation function for weighted inputs of nodes included in a previous layer may be input. A weighted input may be obtained by multiplying an input of nodes included in a previous layer by a weight of a link. A weight may also be referred to as a parameter of the neural network, and updated by training to be described later with reference toFIGS.5and6. A transfer of a weighted input to a subsequent layer may be referred to as propagation. The activation function may include sigmoid, hyperbolic tangent (tanh) and rectified linear unit (ReLU), by which nonlinearity may be formed in the neural network.

Hereinafter, an image segmentation method performed using the image segmentation model300will be described with reference toFIG.2in conjunction withFIG.3.

In operation210, a processor of an electronic device of an example may extract feature data from an input image301based on a feature extraction layer310. The electronic device may extract the feature data from the input image301using the feature extraction layer310through the data propagation described above.

In operation220, the processor of the electronic device may calculate one or more class maps325from the extracted feature data based on a classifier layer320. The electronic device may calculate (or obtain) the class maps325from the feature data using the classifier layer320through the data propagation described above. For example, the electronic device may calculate (or obtain) class maps (e.g., the class maps325) as many as the number of available classes of the classifier layer320. The electronic device may calculate a class map for each available class. An available class described herein may be a class that is classifiable by the classifier layer320, and the number of available classes may indicate the number of classes distinguishable in the classifier layer320. For example, when the number of available classes in the classifier layer320is N, the electronic device may calculate a class map for each of the N available classes. In this example, N may be an integer greater than or equal to 2. That is, the electronic device may calculate the N class maps (e.g., the class maps325). For example, the number of elements in each of the class maps325may be the same as the number of pixels of the input image301. A class map corresponding to one class may be a set of element values indicating probabilities that pixels of the input image301belong to the class.

In operation230, the processor of the electronic device may calculate one or more cluster maps335from the extracted feature data based on a clustering layer330. The electronic device may calculate the cluster maps335from the feature data using the clustering layer330. For example, the electronic device may calculate (or obtain) cluster maps (e.g., the cluster maps335) as many as the number of available clusters of the clustering layer330. The number of available clusters may be the number of clusters that available for clustering in the clustering layer330. For example, when the number of available clusters in the clustering layer330is M, the electronic device may calculate M cluster maps. In this example, M may be an integer greater than or equal to 2. For example, the number of elements in each cluster map may be the same as the number of pixels of the input image301. A cluster map corresponding to one cluster may be a set of element values indicating probabilities that pixels of the input image301belong to the cluster. An object type indicated by each cluster may be unfixed, and an object type indicated by a cluster map may vary depending on a scene.

In an example, as illustrated inFIG.3, the electronic device may separately calculate (or obtain) the class maps325and the cluster maps335using common feature data for the classifier layer320and the clustering layer330.

In operation240, the electronic device may generate image segmentation data309based on the class maps325and the cluster maps335. The electronic device may generate the image segmentation data309by labeling the pixels of the input image301into a class based on the class maps325and a cluster based on the cluster maps335. The image segmentation data309may be data in which the pixels of the input image301are segmented into classes and/or clusters to which the pixels of the input image301belong respectively.

For example, the electronic device may determine a class to be labeled for each pixel of the input image301, using the class maps325in the input image301. The electronic device may extract element values corresponding to the same pixel of the input image301in the class maps325, and determine a class to be labeled for the pixel using the extracted element values. For example, the electronic device may extract probability values for each class with respect to each pixel of the input image301, and label a class of a class map having a highest probability value among the extracted probability values for a corresponding pixel. However, this is merely an example, and the labeling using the class maps325is not limited to the foregoing operations. For reference, the electronic device may label at least a portion of the pixels of the input image301as an object class or a foreground class, and may label a remaining portion of the pixels as a background class.

For another example, the processor of the electronic device may exclude a clustering result that is based on the cluster maps335for pixels labeled based on the class maps325among the pixels of the image segmentation data309. The electronic device may determine that a pixel classified as the background class in the class maps325belongs to a cluster corresponding to one cluster map among the cluster maps335. That is, the electronic device may cluster the remaining pixels, excluding the pixels labeled using the class maps325, using the cluster maps335. The electronic device may extract element values corresponding to the same pixel of the input image301in the cluster maps335for a region classified as the background class, and determine a cluster to be labeled for the pixel using the extracted element values. For example, the electronic device may extract probability values for each cluster with respect to each pixel classified as the background class in the input image301, and label a cluster of a cluster map having a highest probability value among the extracted probability values for a corresponding pixel. However, this is merely an example, and the labeling using the cluster maps335is not limited to the foregoing operations. For example, when two or more cluster maps represent a clustering result of the same or similar elements, the electronic device may temporarily merge clusters indicated by the two or more cluster maps into a single cluster. The electronic device may label the temporarily merged single cluster for pixels classified as the background class. The temporarily merged single cluster in one scene may be separated in another scene.

Instead of labeling a background including a plurality of objects as a single class, the electronic device may distinguish it into various clusters through the clustering described above. The electronic device may extract more features even from an unlabeled background portion by applying, to learning or training, information on identicalness and dissimilarity among various objects. In addition, it is possible to prepare for a background conflict in continual learning of image segmentation.

In an example, the class maps325, the cluster maps335, and the image segmentation data309may have a resolution of the input image301. For example, the resolution of the class maps325, the cluster maps335, and the image segmentation data309may be the same as the resolution of the input image301. The number of elements of the class maps325, the number of elements of the cluster maps335, and the number of elements of the image segmentation data309may be the same as the number of pixels of the input image301.

FIG.4illustrates an example of using a dynamic model based on a complexity of a surrounding environment.

A processor of an electronic device of an example may select a classifier layer from among a plurality of classifier layers based on a position of the electronic device. For example, the electronic device may store the classifier layers in a memory. At least one classifier layer among the classifier layers may have available classes different from those of another classifier layer. The classifier layers may have different numbers of available classes. The electronic device may selectively use one classifier layer among the classifier layers stored in the memory based on the required number of available classes. The electronic device may estimate the position of the electronic device using a global navigation satellite system (GNSS) signal received using a GNSS module. However, the estimation of the position is not limited thereto, and the electronic device may estimate the position using a positional relationship with a base station connected through a communication module and a geographical position of an access point (AP) being connected. The required number of available classes may vary based on a complexity of a surrounding environment that is based on the position of the electronic device.

The electronic device may calculate (or obtain) one or more class maps from extracted feature data using the selected classifier layer. The electronic device may connect the selected classifier layer to a feature extraction layer. The electronic device may calculate (or obtain) a class map for each available class of the selected classifier layer by propagating feature data extracted from an input image using the feature extraction layer to the selected classifier layer.

In operation450, the electronic device may estimate the complexity of the surrounding environment based on the position of the electronic device. The complexity of the surrounding environment may be a degree of how complex the surrounding environment of the electronic device is, and vary based on the number of types of objects that may be potentially present in the surrounding environment. That is, as the complexity of the surrounding environment increases, the required number of available classes may increase. In contrast, as the complexity of the surrounding environment decreases, the required number of available classes may decrease. For example, when the electronic device is provided in a vehicle and the vehicle enters a center of a city, the complexity of the surrounding environment of the electronic device may increase. When the vehicle moves out to the suburbs, the complexity of the surrounding environment of the electronic device may decrease. For another example, when the electronic device is provided in an AR device and the AR device moves indoors, the complexity of the surrounding environment of the electronic device may increase. When the AR device moves outdoors, the complexity of the surrounding environment of the electronic device may decrease. However, examples of the increase and decrease in the complexity of the surrounding environment is not limited to the foregoing.

For example, when the complexity of the surrounding environment increases due to a change in the position of the electronic device while the processor of the electronic device is calculating a class map using a first classifier layer421having a first number (e.g., 100) of available classes, the processor of the electronic device may select a second classifier layer422having a second number (e.g., 1000) of available classes that is greater than the first number of available classes. When the complexity of the surrounding environment decreases due to a change in the position of the electronic device while the processor of the electronic device is calculating a class map using the second classifier layer422having the second number of available classes, the processor of the electronic device may select the first classifier layer421having the first number of available classes that is less than the second number of available classes. Thus, the electronic device may selectively provide labeling that uses classifier layers having various numbers of available classes, using the same feature extraction layer. Although the first classifier layer421and the second classifier layer422have been described above, the electronic device may store two or more classifier layers. The electronic device may select a classifier layer corresponding to an estimated complexity of the surrounding environment from among the two or more classifier layers.

Although an increase or a decrease in the number of available classes of a classifier layer has been described above, examples are not limited thereto. For example, the number of available clusters of a clustering layer may also increase and/or decrease. In this example, the processor of the electronic device may select one clustering layer from among a plurality of clustering layers based on the position of the electronic device. The electronic device may connect the selected clustering layer to the feature extraction layer. The electronic device may calculate (or obtain) one or more cluster maps from extracted feature data using the selected clustering layer. Thus, the electronic device may selectively provide clustering that uses clustering layers having various numbers of available clusters, using the same feature extraction layer.

For example, when the complexity of the surrounding environment increases due to a change in the position of the electronic device while the processor of the electronic device is calculating a cluster map using a first clustering layer having a first number of available clusters, the processor of the electronic device may select a second clustering layer having a second number of available clusters that is greater than the first number of available clusters. When the complexity of the surrounding environment decreases due to a change in the position of the electronic device while the processor of the electronic device is calculating a cluster map using the second clustering layer having the second number of available clusters, the processor of the electronic device may select the first clustering layer having the first number of available clusters that is less than the second number of available clusters Although the first clustering layer and the second clustering layer have been described above, the electronic device may store two or more clustering layers. The electronic device may select a clustering layer corresponding to an estimated complexity of the surrounding environment from among the two or more clustering layers.

Although the example in which the complexity of the surrounding environment varies depending on the position of the electronic device has been mainly described above, examples are not limited thereto. For example, the electronic device may estimate the complexity of the surrounding environment by analyzing a scene of an input image.

FIGS.5and6illustrate examples of training an image segmentation network.

A processor of an electronic device of an example may update one of or a combination of two or more of a feature extraction layer510, a classifier layer520, and a clustering layer530based on training data. For example, the training data may be collected during a movement of the electronic device, but the process is not limited thereto. The training data may include a pair of a training input (e.g., a training image501) and a ground truth label map580. The ground truth label map580may be a map in which a class to which individual pixels of the training image501belongs is labeled. A portion shown as unlabeled in the example ofFIG.5may be a portion in which a background class is labeled. That is, a portion in which an object class is unlabeled may be labeled as the background class.

The processor of the electronic device may update a parameter of the classifier layer520based on a first loss591between a ground truth label of the collected training data and one or more class maps calculated from the classifier layer520. For example, the electronic device may calculate first feature data515aby propagating the training input to the feature extraction layer510. The electronic device may calculate a temporary class map by propagating the first feature data515ato the classifier layer520. The temporary class map may be generated for each available class of the classifier layer520. The electronic device may calculate, as the first loss591, a cross-entropy loss LCEbetween the temporary class map and the ground truth label map580.

The processor of the electronic device may update a parameter of the clustering layer530using a second loss592calculated based on the training image501of the collected training data and an image augmented from the training image501. The electronic device may generate an image set (In, In′, In″) to which various augmentation methods are applied to the training image501.

For example, the electronic device may generate a first augmented image501band a second augmented image501cby augmenting the training image501. The electronic device may generate an augmented image by emphasizing one color component in the training image501. For example, the first augmented image501bmay be an image in which a green component is emphasized, and the second augmented image501cmay be an image in which a blue component is emphasized. However, an augmentation method is not limited to such a color change, and other various image augmentation methods including changing a color tone and/or brightness or preserving a shape component such as an edge of an image may also be used. The electronic device may calculate feature data corresponding to each image by individually propagating the training image501, the first augmented image501b, and the second augmented image501cto the feature extraction layer510. For example, as illustrated inFIG.5, first feature data515amay be data extracted from the training image501, second feature data515bmay be data extracted from the first augmented image501b, and third feature data515cmay be data extracted from the second augmented image501c. Although it is illustrated that two additional images are augmented inFIG.5, the number of augmented images is not limited thereto.

The electronic device may calculate a cluster map corresponding to each feature data by individually propagating the first feature data515a, the second feature data515b, and the third feature data515cto the clustering layer530. For example, the electronic device may generate a first temporary cluster map calculated from the first feature data515a, a second temporary cluster map calculated from the second feature data515b, and a third temporary cluster map calculated from the third feature data515c. The electronic device may calculate the second loss592using temporary cluster maps generated from an original image (e.g., the training image501) and augmented images. For example, the electronic device may calculate, as the second loss592, a loss Lclustercorresponding to a difference among the first temporary cluster map, the second temporary cluster map, and the third temporary cluster map. A difference between temporary cluster maps may include, for example, a difference between vectors each indicating a cluster probability for each pixel, and a difference between data obtained by transforming the vectors (e.g., data obtained by normalizing or transforming vector values). A vector representing a cluster probability for each pixel may be a 1×1×M dimensional vector having, as an element, a probability that each pixel belongs to each cluster for each of W×H pixels in a W×H×M-dimensional cluster map.

The electronic device may update parameters of layers of an image segmentation model, using a total loss that is based on the first loss591and the second loss592. For example, the electronic device may update the parameters until the total loss is less than a threshold value while backpropagating the total loss. The electronic device may update the parameters based on a gradient descent method. However, examples are not limited thereto, and the electronic device may repeat updating the parameters until the total loss converges. The total loss may be, but is not limited to, a weighted sum of the first loss591and the second loss592.

The electronic device may train the image segmentation model such that the first loss591is reduced, and thereby train the classifier layer520with label information of the ground truth label map580. The electronic device may train the image segmentation model such that the second loss592is reduced, and thereby update a parameter of the clustering layer530such that the same or similar clusters are formed from pixel-wise feature vectors Vij, Vij′, and Vij′′ of each image extracted from the feature extraction layer510. That is, through training using the second loss592, the electronic device may group together pixels representing the same or similar objects and/or background in an input image.

Thus, the electronic device may label a cluster for a pixel for which a class is unlabeled by training the parameters of the classifier layer520and the clustering layer530in time series or in parallel. Accordingly, even when only some classes in a training image are labeled (annotated), the electronic device may efficiently perform training. For reference, the second loss592ofFIG.5may be an example clustering loss, and a loss used for training the clustering layer530is not limited thereto.

FIG.6illustrates an example of training performed when a target class increases while a classifier layer is being upgraded.

For example, through training in a similar way described above with reference toFIG.5, an electronic device of an example may update parameters of a feature extraction layer610, a classifier layer620a, and a clustering layer630. For example, the number of available classes of the classifier layer620amay be N including, for example, a road, a sidewalk, a vehicle, and a background (N=4). After the training described above is completed for a current available class, an available class may be added.

A processor of the electronic device may train the previously constructed classifier layer620aand a new classifier layer620bincluding a layer corresponding to a class to be added, based on collected training data. The electronic device may combine a cluster map672corresponding to a new label (e.g., a traffic light) among cluster maps calculated using a previously trained image segmentation model and an output of a previously constructed classifier layer (e.g., the classifier layer620a) to generate new ground truth class data680and collect it as training data.

For example, the processor of the electronic device may obtain a clustering result corresponding to a class to be added from one or more cluster maps calculated by the previously constructed clustering layer630.

In the example ofFIG.6, the electronic device may generate cluster maps U as many as the number of available clusters using the clustering layer630. An input image601may have a resolution of H×W, in which H denotes the number of pixels corresponding to the height and W denotes the number of pixels corresponding to the width. The electronic device may generate cluster maps of H×W as many as M available clusters. A set of cluster maps may also be represented as cluster data U, and U [H×W×M] may represent cluster data635including the cluster maps of H×W as many as the M available clusters. Similarly, the electronic device may generate class maps as many as the number of available classes by using a classifier layer. For example, the electronic device may generate class maps of H×W as many as N available classes. In this example, N and M may each be an integer greater than or equal to 1. A set of class maps may also be represented as class data O, and O [H×W×N] may represent class data625aincluding the class maps of H×W as many as the N available classes.

The electronic device may select the cluster map672corresponding to the class to be added from among one or more cluster maps. For example, the electronic device may extract probabilities (e.g., cluster probabilities) that a pixel671corresponding to the class to be added among pixels labeled as a background class belong to each cluster. The electronic device may determine, as a cluster corresponding to the class to be added, a cluster of the cluster map672indicating the highest cluster probability among the cluster probabilities of the pixel671corresponding to the class to be added. However, examples are not limited thereto, and the electronic device may determine, as the cluster corresponding to the class to be added, clusters having probability values in a preset upper rank in descending order with respect to the pixel671corresponding to the class to be added. For example, the electronic device may select the cluster map672in which the pixel671corresponding to a traffic light, which is the class to be added, has a high probability from among the M cluster maps. For example, the pixel671corresponding to the class to be added in the image may be set by a user and/or an expert. Since only the class to be added needs to be annotated, the cost of constructing a dataset may be reduced.

The electronic device may update a parameter of the new classifier layer620bbased on a third loss between an obtained clustering result and a class map calculated using the new classifier layer620b. For example, the electronic device may calculate new class data625bby propagating the feature data extracted through the feature extraction layer610to the new classifier layer620b.

The new class data625bmay include N+1 class maps corresponding to the existing N available classes and an available class to be newly added. In the example ofFIG.6, the N+1 class maps of H×W may be generated. However, this is merely an example in which one available class is added. For another example, when k available classes are added, N+k new class maps may be generated. In this example, k may be an integer greater than or equal to 1. The electronic device may combine the class data625aO [H×W×N] which is an output of the previous classifier layer620aand the cluster map672corresponding to the class to be added, to generate the new ground truth class data680O′ [H×W× (N+1)].

The electronic device may calculate, as a third loss693, a loss (e.g., a distillation loss Ldistill) between an output of the new classifier layer620band the new ground truth class data680O′ [H×W× (N+1)]. The electronic device may calculate the third loss693only for the pixel671corresponding to a portion (e.g., a traffic light) corresponding to the class to be added in the new ground truth class data680O′ [H×W× (N+1)] and the class data625bwhich is an output of the new classifier layer620b, excluding a remaining portion from the calculation of the third loss693. The electronic device may iteratively update a parameter of the new classifier layer620bsuch that the third loss693is reduced. When the update based on the third loss693is completed, an initial parameter of the new classifier layer620bmay be set. In addition, the electronic device may calculate a total loss by calculating e a sum of a first loss (e.g., the first loss591ofFIG.5), a second loss (e.g., the second loss592ofFIG.5), and the third loss693, and iteratively train at least one layer of the image segmentation model until the total loss is reduced to a threshold or converges. Thus, using the third loss693, the electronic device may reduce a learning (or training) time, reduce a required resource, and prevent deterioration of learning (or training) that may occur due to a collision between a newly added class and a previously learned background. The electronic device may use only a label for a class to be added, enabling continual learning while maintaining a learned knowledge of a previous class.

Although the example of using an output of the previously constructed classifier layer620afor a previously constructed available class among available classes of the new classifier layer620bhas been described above, examples are not limited thereto. The electronic device may calculate the third loss693by using an output of the previously constructed clustering layer630only for a newly added available class and using a prepared ground truth label map for an existing available class.

The electronic device may perform an update while maintaining parameters of the feature extraction layer610and the clustering layer630and replacing a previous one (e.g., the classifier layer620a) with the new classifier layer620b. However, the process is not limited thereto, and the electronic device may update all the parameters of the feature extraction layer610, the clustering layer630, and the new classifier layer620b. In this case, the number of available clusters of the clustering layer630may be reduced by the number of added classes. For example, when the number of available classes of the new classifier layer620bis N+k, the electronic device may change the image segmentation model to have a new clustering layer having M−k available clusters.

FIG.7illustrates an example of an electronic device.

Referring toFIG.7, an electronic device700may include an image acquirer710, a processor720, a memory730, and a display740.

The image acquirer710may obtain an image of a scene around the electronic device700. In an example, the image acquirer710may obtain a scene around (e.g., in front) of a device (e.g., a vehicle and/or an AR device) in which the electronic device700is provided. For example, the image acquirer710may include an imaging module that generates an image by capturing an image of such a surrounding scene. The image acquirer710may include a camera sensor, a light detection and ranging (lidar) sensor, a radio detection and ranging (radar) sensor, and an ultrasonic sensor. For example, the electronic device700may generate a color image through sensing data of the camera sensor. For example, the electronic device700may generate a lidar image through sensing data of the lidar sensor. For example, the electronic device700may generate a radar scan image through sensing data of the radar sensor. For example, the electronic device700may generate an ultrasonic image through sensing data of the ultrasonic sensor. However, the configuration is not limited thereto, and the image acquirer710may include a communicator. For example, the communicator of the electronic device700may receive an image from another imaging device disposed outside the electronic device700by wire and/or wirelessly.

The processor720may extract feature data from an input image based on a feature extraction layer. The processor720may calculate (or obtain) one or more class maps from the extracted feature data based on a classifier layer. The processor720may calculate (or obtain) one or more cluster maps from the extracted feature data based on a clustering layer. The processor720may generate image segmentation data based on the class maps and the cluster maps. However, operations of the processor720are not limited thereto, and the processor720may also perform the operations described above with reference toFIGS.1through6.

The memory730may temporarily or permanently store data required for image segmentation. For example, the memory730may store therein the input image, training data, an image segmentation model, the feature extraction layer, the classifier layer, the clustering layer, a parameter of each layer, and an inference result (e.g., image segmentation data).

The display740may visually output the image segmentation data. For example, the display740of the electronic device700provided in a vehicle may visually display information associated with traveling of the vehicle that is generated using the image segmentation data. The vehicle may be an autonomous vehicle or a vehicle supporting advanced driver-assistance systems (ADAS).

For another example, the display740of the electronic device700provided in an AR providing device may visually provide a user with an AR content generated using the image segmentation data.

In an example, the electronic device700may maintain a knowledge of a previously learned class when adding a new class, by performing training and inference using both the classifier layer and the clustering layer. In addition, the electronic device700may train the feature extraction layer such that it outputs significant and rich feature vectors even for unlabeled pixels (e.g., background) in addition to labeled pixels.

The electronic device700may also be applied to devices for content-based image retrieval, machine vision, medical imaging, video surveillance, and the like that use image segmentation, in addition to the vehicle and the AR device.

The electronic device700may perform continual learning that enables sequential adaptation to training data that changes in various ways without a need to retrain a network with all the training data. The electronic device700may perform annotation and training only on a newly added class, and it is thus possible to reduce the cost and time used. In addition, the electronic device700may perform clustering on a background, and may thus provide a pseudo-labeling result even for an unlabeled portion. Through distillation loss-based learning, it is possible to prevent a knowledge conflict between a previously learned background portion and a portion corresponding to a newly added class in a learning or training process.

The electronic device, and other devices, apparatuses, units, modules, and components described herein with respect toFIGS.1and3-7are implemented by or representative of hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term “processor” or “computer” may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

The methods illustrated inFIGS.1-7that perform the operations described in this application are performed by computing hardware, for example, by one or more processors or computers, implemented as described above executing instructions or software to perform the operations described in this application that are performed by the methods. For example, a single operation or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors, or a processor and a controller, and one or more other operations may be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may perform a single operation, or two or more operations.

Instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above may be written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the one or more processors or computers to operate as a machine or special-purpose computer to perform the operations that are performed by the hardware components and the methods as described above. In one example, the instructions or software include machine code that is directly executed by the one or more processors or computers, such as machine code produced by a compiler. In another example, the instructions or software includes higher-level code that is executed by the one or more processors or computer using an interpreter. The instructions or software may be written using any programming language based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations that are performed by the hardware components and the methods as described above.

The instructions or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above, and any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of a non-transitory computer-readable storage medium include read-only memory (ROM), random-access programmable read only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, a card type memory such as multimedia card micro or a card (for example, secure digital (SD) or extreme digital (XD)), magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and any other device that is configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed over network-coupled computer systems so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by the one or more processors or computer.

While this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and details may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein are to be considered in a descriptive sense only, and not for purposes of limitation. Descriptions of features or aspects in each example are to be considered as being applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, and/or replaced or supplemented by other components or their equivalents.

Therefore, the scope of the disclosure is defined not by the detailed description, but by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure.