Patent ID: 12212851

DETAILED DESCRIPTION

Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. In this regard, the present embodiments may have different forms and should not be construed as being limited to the descriptions set forth herein. Accordingly, the embodiments are merely described below, by referring to the figures, to explain aspects of the present description. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of,” when preceding a list of elements, modify the entire list of elements and do not modify the individual elements of the list.

As the present disclosure allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. Advantages, features, and a method of achieving the same will be specified with reference to the embodiments described below in detail together with the attached drawings. However, the present disclosure may have different forms and should not be construed as being limited to the descriptions set forth herein.

The embodiments of the present disclosure will be described below in more detail with reference to the accompanying drawings. Those elements that are the same or are in correspondence are rendered the same reference numeral regardless of the figure number, and redundant explanations are omitted.

It will be understood that although the terms “first”, “second”, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These elements are only used to distinguish one element from another. Herein, singular expressions, unless defined otherwise in contexts, include plural expressions. In the embodiments below, it will be further understood that the terms “comprise” and/or “have” used herein specify the presence of stated features or elements, but do not preclude the presence or addition of one or more other features or elements. In the drawings, for convenience of description, sizes of elements may be exaggerated or contracted. In other words, since sizes and thicknesses of elements in the drawings are arbitrarily illustrated for convenience of explanation, the following embodiments are not limited thereto.

FIG.1is a schematic structural diagram of a weapon system, according to an embodiment.

When an image acquisition device included in the weapon system according to an embodiment has failed to track a first object which is a tracking target object, the weapon system may determine a reference object in an image and set an image capturing range and direction based on the reference object, and recognize the first object around the reference object again.

The weapon system according to the embodiment may include a control apparatus100, an image acquisition device200, and a firing device300, as illustrated inFIG.1.

When the image acquisition device200has failed to track the first object, which is a tracking target object, the control apparatus100according to an embodiment may determine a reference object in an image and set an image capturing range and direction of the image acquisition device200based on the reference object, and recognize the first object around the reference object again.

In addition, the control apparatus100according to the embodiment may control the firing device300based on tracking information about the first object, the tracking information being generated based on the image acquired using the image acquisition device200. For example, the control apparatus100may control the firing device300to perform firing at the first object based on location information about the first object. However, this is an example, and the present disclosure is not limited thereto.

FIG.2is a schematic structural diagram of a control apparatus100, according to an embodiment. Referring toFIG.2, the control apparatus100according to an embodiment may include a communication interface110, a first processor120, a memory130, and a second processor140. Although not illustrated in the drawings, the control apparatus100may further include an input/output device, a program storage, or the like, according to an embodiment.

The communication interface110may include a device including hardware or software components required by the control apparatus100to transmit or receive a signal such as a control signal or a data signal through a wired/wireless connection with a tactical unit such as a moving vehicle (not shown), a vessel (not shown), an armored vehicle (not shown), and the like. The communication interface110may include at least one of a digital modem, a radio frequency (RF) modem, a WiFi chip, and related software and/or firmware.

The first processor120may perform tracking of the first object based on the image acquired by the image acquisition device200. The first processor120may also control the image acquisition device200and/or the firing device300based on the tracking information about the first object. In this process, the first processor120may control a series of processes of confirming a location of the first object in the image by using a trained artificial neural network.

A processor herein may refer to, for example, a data processing device that is embedded in a hardware component and has a physically structured circuit to perform a function expressed as a code or a command included in a program. Examples of the data processing device embedded in a hardware component may encompass processing devices such as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), and the like, but the present disclosure is not limited thereto.

The memory130performs a function of temporarily or permanently storing data processed by the control apparatus100. The memory may include a magnetic storage medium or a flash storage medium, but the scope of the present disclosure is not limited thereto. For example, the memory130may temporarily and/or permanently store an image acquired using the image acquisition device200. Also, the memory130may temporarily and/or permanently store data (e.g., coefficients) constituting the trained artificial neural network. However, this is an example, and the present disclosure is not limited thereto.

The second processor140may refer to a device that performs an operation under the control by the above-described first processor120. In this case, the second processor140may be a device having a higher arithmetic capability than the above-described first processor120. For example, the second processor140may be configured as a graphics processing unit (GPU). However, this is an example, and the present disclosure is not limited thereto. In an embodiment, a plurality of second processors140or a single second processor140may be included. According to an embodiment, the second processor140may be included in or implemented as a sub-processor of the first processor120.

According to an embodiment, the image acquisition device200may include various types of devices for acquiring an image of the surroundings according to the control by the control apparatus100described above. According to an embodiment, the image acquisition device200may be a device capable of performing panning, tilting or zooming such as a pan/tilt/zoom (PTZ) camera including one or more complementary metal oxide semiconductor (CMOS) image sensors. The first processor120and/or the second processor140may determine at least one of a pan value, a tilt value, and a zoom value of the image acquisition device200according to a moving direction and/or a location of the first object which is a tracking target object, and provide the same to the image acquisition device200, and the image acquisition device200may acquire an appropriate image, accordingly. However, this is an example, and the present disclosure is not limited thereto.

While the image acquisition device200and the firing device300are illustrated as an integrated configuration InFIG.1, the present disclosure is not limited thereto, and the image acquisition device200and the firing device300may be provided independently of each other.

The firing device300may include various types of devices for performing firing according to the control by the control apparatus100described above, according to an embodiment. For example, the firing device300may be a device that fires bullets and/or shells toward a target.

In an embodiment, the control apparatus100may generate location information about the first object based on the image acquired using the image acquisition device200, and transfer the location information about the first object to the firing device300, so that the firing device300may perform firing at the first object. However, this is an example, and the present disclosure is not limited thereto.

Hereinafter, description will focus on an operation of the control apparatus100, according to an embodiment.

In the present disclosure, an “artificial neural network” may include a neural network trained with learning data according to uses, and may refer to an artificial neural network trained by a machine learning or deep learning technique. A structure of the above neural network will be described with reference toFIG.3, and learning data of the artificial neural network will be described with reference toFIG.4, and input data and output data of the artificial neural network will be described with reference toFIG.5.

The control apparatus100may train an artificial neural network for recognizing a certain object in an image by using learning data, according to an embodiment.

FIG.3is a diagram for describing an example structure of an artificial neural network trained by the control apparatus100, according to an embodiment.

The artificial neural network may include an artificial neural network according to a convolutional neural network (CNN), as illustrated inFIG.3, according to an embodiment. The CNN model may be a layer model used to alternately perform a plurality of computational layers (convolutional layer, pooling layer) to finally extract features of input data.

The control apparatus100may establish or train an artificial neural network model by processing learning data according to a supervised learning technique, according to an embodiment. A method, performed by the control apparatus100, of training an artificial neural network will be described later with reference toFIG.4.

According to an embodiment, the control apparatus100may train an artificial neural network, by using a plurality of learning data, by repeatedly performing an operation of updating a weight of each layer and/or each node such that an output value generated by inputting any one piece of input data into the artificial neural network is close to a value marked on the learning data.

The control apparatus100may update a weight (or coefficient) of each layer and/or each node according to a back propagation algorithm, according to an embodiment.

The control apparatus100may generate a convolution layer for extracting feature values of input data and a pooling layer for configuring a feature map by combining the extracted feature values, according to an embodiment.

In addition, the control apparatus100may generate, by combining the generated feature maps, a fully connected layer that prepares to determine a probability that input data corresponds to each of a plurality of items, according to an embodiment.

The control apparatus may generate an output layer including an output corresponding to input data, according to an embodiment.

In the example illustrated inFIG.3, input data is divided into 5×7 blocks, a 5×3 unit block is used to generate a convolution layer, and a 1×4 or 1×2 unit block is used to generate a pooling layer. However, this is an example, and the present disclosure is not limited thereto. Accordingly, the type of input data and/or the size of each block may be variously configured.

The artificial neural network described above may be stored in the memory130described above in the form of coefficients of a function defining the type of the artificial neural network model, a coefficient of at least one node constituting the artificial neural network, a weight of the node, and a relationship between a plurality of layers constituting the artificial neural network. The structure of the artificial neural network may also be stored in the memory130in the form of source code and/or a program.

The types and/or structures of the artificial neural network described inFIG.3are examples, and the present disclosure is not limited thereto. Therefore, artificial neural networks of various types of models may correspond to the “artificial neural networks” described throughout the specification.

FIG.4is a diagram for describing a method of training an artificial neural network by using a plurality of learning data.FIG.5is a diagram for describing a process of outputting location information about an object by using a trained artificial neural network, according to an embodiment.

Referring toFIGS.4and5. an artificial neural network520may refer to a neural network that is trained (or learns) a correlation between images included in each of a plurality of training data510and information about objects in the images, according to an embodiment.

Accordingly, the artificial neural network520may be a neural network trained to output information532about an object included in an image531according to input of the image531as illustrated inFIG.5. The information532about the object may include information for recognizing the object in the image531(e.g., information about the shape of the object, information about the class of the object, etc.).

Each of the plurality of learning data510according to the embodiment may include an image including an object and information about the object included in that image. For example, first training data511may include an image511A including an object and information511B about the object in the image511A. For example, when an object is a car, the image511A may include a car as illustrated, and the information511B about the object may include information about the shape of the car. According to another embodiment, the information511B about the object may further include information about the class (or type) of the object.

Similarly, second training data512and third training data513may also include an image including an object and information about the object included in the image, respectively. Here, the image511A of the first training data511, the image of the second training data512and the image of the third training data513may form a video or a video clip captured by the same image acquisition device200.

Hereinafter, it will be described on the basis of the assumption that learning of the artificial neural network has been completed according to the above-described process.

The control apparatus100may identify a location of an object in an image including the object by referring to the image including the object and information about the object included in the image, according to an embodiment.

In addition, the control apparatus100may obtain a location of an object in an actual space of the object based on a pan value, a tilt value, and a zoom value at a time when the image acquisition device200acquires an image including the object and a location of the object in the image, according to an embodiment.

The control apparatus100according to the embodiment may track the first object, which is the tracking target object, based on the image acquired by the image acquisition device200. In this case, “tracking” the first object may indicate determining a location of the first object that is changed according to a movement of the first object in real time, and controlling the image acquisition device200and/or the firing device300accordingly, according to an embodiment.

The control apparatus100may track the first object by using the artificial neural network520trained according to the process described with reference toFIGS.4and5. For example, the control apparatus100may input an image such as the image511A acquired by the image acquisition device200to the artificial neural network520to generate object information for the image.

In addition, as described above, the control apparatus100may determine the location of the object in the acquired image by referring to the image acquired by the image acquisition device200and the information about the object generated by the artificial neural network520, as described above. The control apparatus100may also obtain a location of an object in an actual space based on a pan value, a tilt value, and a zoom value at a time when the image acquisition device200acquires an image and the location of the object in the image.

However, the object tracking method using the artificial neural network trained as described above is an example, and the present disclosure is not limited thereto, and various object tracking methods may be used without limitation to the extent that the methods are consistent with the present disclosure.

According to an embodiment, the control apparatus100may track the first object by using information on the first object, the information being collected in the process of tracking the first object, in addition to the information about the object, generated by the artificial neural network520described above. For example, the control apparatus100may track the first object by using class information about the first object, color information about the first object, movement pattern information about the first object, and the like. However, this is an example, and the present disclosure is not limited thereto.

According to an embodiment, the control apparatus100may detect an event in which tracking of the first object has failed, from a first image acquired by the image acquisition device200. For example, when the first object is obscured by another object and disappears from the first image, the control apparatus100may determine that an event in which tracking of the first object has failed is detected. Also, even when the first object, which the control apparatus100was tracking, is covered up, the control apparatus100may determine that an event in which tracking of the first object has failed is detected. However, the above-described conditions of a tracking failure are examples, and the present disclosure is not limited thereto.

According to an embodiment, the control apparatus100may determine a reference object which is a reference for controlling the image acquisition device200in the first image according to the detection of the above-described event. The determined reference object may be a reference object for determining an image capturing range and an image capturing direction of the image acquisition device200.

FIGS.6and7are diagrams for describing a process, performed by a control apparatus, according to an embodiment, of determining a reference object.

As illustrated inFIG.6, when at least a portion of a first object611is obscured by a second object612that is different from the first object611, the control apparatus100may set the second object612covering the first object611, as a reference object, according to an embodiment.

Meanwhile, as illustrated inFIG.7, the control apparatus100may determine, as a reference object, a second object622among objects placed in a moving direction V1of a first object621, according to an embodiment. The control apparatus100may determine the moving direction V1of the first object621based on at least one image acquired by the image acquisition device200earlier than a first image.

The control apparatus100may adjust at least one of an image capturing range and an image capturing direction of the image acquisition device200based on at least one of a size and a location of the reference object determined in the above-described process, in the first image, according to an embodiment.

To this end, the control apparatus100may determine a region corresponding to the reference object in the first image, according to an embodiment. For example, the control apparatus100may determine a region occupied by the reference object in the first image as the region corresponding to the reference object.

Then, the control apparatus100may adjust at least one of the image capturing range and the image capturing direction based on a relative locational relationship between the region corresponding to the reference object and a boundary line of the first image. The “boundary line” of the first image may refer to an outermost portion of the first image, that is, a line dividing the first image from the other portion in the first image.

For example, when the region corresponding to the reference object abuts the boundary line of the first image, the control apparatus100may adjust an image capturing range and an image capturing direction of the image acquisition device200such that the image capturing range of the image acquisition device200is extended and the reference object is positioned in a center portion of a second image.

When the region corresponding to the reference object does not abut the boundary line of the first image, the control apparatus100may adjust the image capturing direction, but not the image capturing range, such that the reference object is positioned in the center portion of the second image.

FIG.8illustrates an example of a first image630acquired in a situation illustrated inFIG.6. Hereinafter, embodiments will be described with reference toFIGS.6through8together.

When normal tracking of the first object611or621is performed using the control apparatus100, the image capturing range of the image acquisition device200may be a range613illustrated inFIG.6or a range623illustrated inFIG.7. Accordingly, in the embodiment ofFIG.6, the image acquisition device200may acquire the first image630illustrated inFIG.8.

When the control apparatus100detects an event in which tracking of the first objects611and621is failed, the control apparatus100may modify the image capturing range and the image capturing direction to a range614illustrated inFIG.6or a range624illustrated inFIG.7. The ranges614and624may include the reference object at a center thereof.

Referring toFIG.8, when a region631corresponding to the reference object and boundary lines632and633of the first image abut on each other, the control apparatus100according to an embodiment may adjust the image capturing range and image capturing direction of the image acquisition device200such that the image capturing range of the image acquisition device200is extended and the reference object is located in a center of the second image. The control apparatus100may change the image capturing range and the image capturing direction to correspond to the range614illustrated inFIG.6as described above.

When changing the image capturing range and the image capturing direction, the control apparatus100may adjust the image capturing range such that a ratio of a region corresponding to the reference object in the second image exceeds a certain first threshold ratio and is less than a certain second threshold ratio. The “second image” may refer to an image acquired by the image acquisition device200in a state in which at least one of the image capturing range and the image capturing direction is adjusted.

The control apparatus100may recognize, in the second image, the first object which is a tracking target object, according to an embodiment.

FIG.9is a diagram for describing a process, performed by a control apparatus100, of recognizing a first object from a second image, according to an embodiment.

The control apparatus100may maintain the image capturing range and the image capturing direction of the image acquisition device200until the first object is recognized (or recognized again) in the second image640, according to an embodiment.

Also, the control apparatus100may set at least a partial region of the second image640as a region of interest in which the first object is expected to appear, according to an embodiment. For example, the control apparatus100may set a region642of the second image640other than the region641corresponding to the second object as a region of interest.

The control apparatus100may recognize the first object in the region of interest642by using a trained artificial neural network. The trained artificial neural network may include a neural network trained to output information about an object included in an image according to the input of the image as described above.

According to an embodiment, the control apparatus100may recognize the first object in the region of interest642by using collected information on the first object together with the artificial neural network. For example, the control apparatus100may also recognize (or detect) the first object by comparing class information about the first object, color information about the first object, and movement pattern information about the first object, etc., which are collected with respect to a plurality of objects existing in the region of interest642. The control apparatus100may use an artificial neural network as an aid. For example, the control apparatus100may generate information (e.g., class information) about objects in the region of interest642by using an artificial neural network. However, this is an example, and the present disclosure is not limited thereto.

When the first object is recognized in the second image640according to the above-described process, the control apparatus100may initiate tracking of the first object by referring to a location of the first object in the second image640, according to an embodiment. Thus, the control apparatus100may change the image capturing range and the image capturing direction of the image acquisition device200again to correspond to a range643illustrated inFIG.9. The control apparatus100may also change the image capturing range and direction of the image acquisition device200in real time according to a movement of the first object.

As described above, according to the present disclosure, re-tracking and accurate tracking of an object may be performed by controlling the image capturing range and the image capturing direction of the image acquisition device200according to situations.

FIG.10is a flowchart of a method, performed by the control apparatus100, of controlling the image acquisition device200, according to an embodiment. Hereinafter, embodiments will be described with reference toFIGS.1through9together, and description of repeated details will be omitted.

The control apparatus100may train an artificial neural network for recognizing a first object in an image by using learning data (S1010). The artificial neural network may include an artificial neural network according to a convolutional neural network (CNN), as illustrated inFIG.3.

The control apparatus100may detect an event in which tracking of the first object has failed, from a first image acquired by the image acquisition device200, according to an embodiment (S1020), For example, when the first object is obscured by another object and disappears from the first image, the control apparatus100may determine that an event in which tracking of the first object has failed is detected. Also, even when the first object, which the control apparatus100was tracking, is covered up, the control apparatus100may determine that an event in which tracking of the first object has failed is detected.

The control apparatus100may determine a reference object which is a reference for controlling the image acquisition device200in the first image according to the detection of the above-described event, according to an embodiment (S1030). The determined reference object may be a reference object for determining an image capturing range and an image capturing direction of the image acquisition device200. The control apparatus100may adjust at least one of an image capturing range and an image capturing direction of the image acquisition device200based on at least one of a size and a location of the reference object determined in the above-described process, in the first image, according to an embodiment (S1040).

To this end, the control apparatus100may determine a region corresponding to the reference object in the first image. For example, the control apparatus100may determine a region occupied by the reference object in the first image as the region corresponding to the reference object.

Then, the control apparatus100may adjust at least one of the image capturing range and the image capturing direction based on a relative locational relationship between the region corresponding to the reference object and a boundary line of the first image.

For example, when the region corresponding to the reference object abuts the boundary line of the first image, the control apparatus100may adjust an image capturing range and an image capturing direction of the image acquisition device200such that the image capturing range of the image acquisition device200is extended and the reference object is positioned in a center portion of a second image.

When the region corresponding to the reference object does not abut the boundary line of the first image, the control apparatus100according to an embodiment may adjust only the image capturing direction such that the reference object is positioned in the center portion of the second image.

The control apparatus100may recognize, in the second image, the first object which is a tracking target object, according to an embodiment (S1050). The control apparatus100may maintain the image capturing range and the image capturing direction of the image acquisition device200until the first object is recognized (or recognized again) in the second image640.

Also, the control apparatus100may set at least a partial region of the second image640as a region of interest in which the first object is expected to appear. For example, the control apparatus100may set a region642of the second image640other than the region641corresponding to the second object as a region of interest.

The control apparatus may recognize the first object in the region of interest642by using a trained artificial neural network.

When the first object is recognized in the second image640according to the above-described process, the control apparatus100may initiate tracking of the first object by referring to a location of the first object in the second image640, according to an embodiment (S1060). Thus, the control apparatus100may change the image capturing range and the image capturing direction of the image acquisition device200again to correspond to a range643illustrated inFIG.9.

As described above, according to the present disclosure, re-tracking and accurate tracking of an object may be performed by controlling the image capturing range and the image capturing direction of the image acquisition device200according to situations.

The above-described embodiments according to the present disclosure may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium may be for storing a program executable by a computer. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like.

The computer program may be specifically designed and configured for the present disclosure or may be known and used by those skilled in the art of computer software. Examples of the computer program may include not only machine code generated by a compiler, but also advanced language code that can be executed by a computer using an interpreter or the like.

The specific implementations described in the present disclosure are only examples, and do not limit the scope in any way. For brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. Furthermore, the connecting lines, or connectors between the components illustrated in the various figures presented are intended to represent exemplary functional relationships and/or physical or logical couplings between the various elements. It should be noted that many alternative or additional functional relationships, physical connections or logical connections may be present in a practical device. Moreover, no item or component is essential to the practice unless the element is specifically described as “essential” or “critical.”

According to the present disclosure, a tracking target object may be tracked again by appropriately controlling an image acquisition device in a situation of a failure in tracking the tracking target object.

In particular, according to the present disclosure, in a situation of a failure in tracking a tracking target object, by adjusting at least one of an image capturing direction and an image capturing range of the image acquisition device based on a reference object as a reference, an image including an object with a high probability to appear again may be acquired, and the furthermore, the tracking target object may be detected again.

In addition, according to the present disclosure, the tracking target object may be accurately tracked, thereby enabling an weapon system to accurately fire at the tracking target object.

It should be understood that embodiments described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each embodiment should typically be considered as available for other similar features or aspects in other embodiments. While one or more embodiments have been described with reference to the figures, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the disclosure as defined by the following claims.