Patent ID: 12260666

DETAILED DESCRIPTION

Hereafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. However, it is to be noted that the present disclosure is not limited to the embodiments but can be embodied in various other ways. Also, the accompanying drawings are provided to help easily understand the embodiments of the present disclosure and the technical conception described in the present disclosure is not limited by the accompanying drawings. In the drawings, parts irrelevant to the description are omitted for the simplicity of explanation, and the size, form and shape of each component illustrated in the drawings can be modified in various ways. Like reference numerals denote like parts through the whole document.

Suffixes “module” and “unit” used for components disclosed in the following description are merely intended for easy description of the specification, and the suffixes themselves do not give any special meaning or function. Further, in the following description of the present disclosure, a detailed explanation of known related technologies may be omitted to avoid unnecessarily obscuring the subject matter of the present disclosure.

Throughout this document, the term “connected to (contacted with or coupled to)” may be used to designate a connection or coupling of one element to another element and includes both an element being “directly connected to (contacted with or coupled to)” another element and an element being “electronically connected to (contacted with or coupled to)” another element via another element. Further, through the whole document, the term “comprises or includes” and/or “comprising or including” used in the document means that one or more other components, steps, operation and/or existence or addition of elements are not excluded in addition to the described components, steps, operation and/or elements unless context dictates otherwise.

Further, in describing components of the present disclosure, ordinal numbers such as first, second, etc. can be used only to differentiate the components from each other, but do not limit the sequence or relationship of the components. For example, a first component of the present disclosure may also be referred to as a second component and vice versa.

FIG.1is a block diagram illustrating a configuration of an image-based animal object condition identification apparatus according to an embodiment of the present disclosure.

Referring toFIG.1, an image-based animal object condition identification apparatus100includes a communication module110, a memory120and a processor130and may further include a database140. The image-based animal object condition identification apparatus100receives images from a plurality of CCTVs installed at a shed in real time, detects an animal object by using the received images and detects the condition of the animal based on the animal detection information.

To this end, the image-based animal object condition identification apparatus100may be implemented with a computer or portable device which can access a server or another device through a network. Herein, the computer may include, for example, a notebook, a desktop and a laptop equipped with a WEB browser. The portable devices may be, for example, a wireless communication device that ensures portability and mobility and may include all kinds of handheld-based wireless communication devices such as various smart phones, tablet PCs, smart watches, and the like.

The term “network” refers to a connection structure that enables information exchange between nodes such as devices, servers, etc. and includes LAN (Local Area Network), WAN (Wide Area Network), Internet (WWW: World Wide Web), a wired or wireless data communication network, a telecommunication network, a wired or wireless television network, and the like. Examples of the wireless data communication network may include 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, ultrasonic communication, VLC (Visible Light Communication), LiFi, and the like, but may not be limited thereto.

The communication module110receives images of an object from one or more cameras. Herein, the object may include various classes of animal objects such as cows, pigs and dogs. The communication module110may include hardware and software required to transmit and receive a signal, such as a control signal or a data signal, through wired/wireless connection with other network devices.

The memory120stores therein a program configured to extract animal detection information from the images received through the communication module110. Herein, the program configured to extract animal detection information extracts continuous animal detection information of each object by inputting the received images into an animal detection model that is trained based on learning data composed of animal images. Also, the program extracts animal condition information by inputting the continuous animal detection information of each object into an animal condition identification model constructed based on learning data in which animal condition information is matched with each class of each animal object. Details of the animal detection information and animal condition information will be described later.

Herein, the memory120may collectively refer to a non-volatile storage device that retains information stored therein even when power is not supplied and a volatile storage device that requires power to retain information stored therein. The memory120may function to temporarily or permanently store data processed by the processor130. The memory120may include magnetic storage media or flash storage media in addition to the volatile storage device that requires power to retain information stored therein, but the present disclosure is not limited thereto.

The processor130executes the program configured to extract the animal condition information stored in the memory120and outputs the animal condition information about the object as a result of execution.

In an example, the processor130may be implemented as a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA), but the scope of the present disclosure is not limited thereto.

The database140may store therein images taken with the cameras and received through the communication module110or various data for training of the animal condition identification model. In particular, images taken with the respective cameras installed at each shed may be distinguished and separately stored in the database140. Also, the database140accumulatively stores the animal detection information and animal condition information extracted by the animal condition information extraction program, and the animal detection information and animal condition information can be used in various applications for monitoring an abnormal condition of an animal.

FIG.2is a block diagram provided to explain a process of outputting animal condition information by the image-based animal object condition identification apparatus according to an embodiment of the present disclosure.

Referring toFIG.2, the image-based animal object condition identification apparatus100extracts continuous animal detection information210of each object by inputting received images10into an animal detection model600that is trained based on learning data composed of animal images. Then, the image-based animal object condition identification apparatus100determines predetermined animal condition information310for each class of each animal object by inputting the continuous animal detection information210of each object into an animal condition identification model30. Herein, the animal detection information210is extracted from n number of continuous entire images including at least one animal object, and includes n number of continuous object images211and n number of continuous object detection data corresponding to the respective object images211.

FIG.3Ais provided to explain animal detection information extracted by an animal detection model of the image-based animal object condition identification apparatus according to an embodiment of the present disclosure.

FIG.3BandFIG.3Care provided to explain an animal condition identification model of the image-based animal object condition identification apparatus according to an embodiment of the present disclosure.

FIG.4andFIG.5are provided to explain animal detection information extracted by the animal detection model of the image-based animal object condition identification apparatus according to an embodiment of the present disclosure.

Hereafter, the animal detection model600that generates the animal detection information210will be described with reference toFIG.3A,FIG.4andFIG.5.

Referring toFIG.3A, the animal detection information210is extracted from n number of continuous entire images200including at least one animal object. That is, the animal detection information210includes the n number of continuous object images211and n number of continuous object detection data212corresponding to the respective object images211.

As shown inFIG.4andFIG.5, the object detection data212refer to information about a bounding box (rbbox) created to be suitable for an animal object detected from the n number of continuous entire images200. That is, the object detection data212include coordinates (xc, yc) of a central point of the bounding box, a width (w) of the bounding box, a length (h) of the bounding box and a rotational angle (theta) of the bounding box with respect to a reference axis.

Also, the object detection data212refer to information indicating keypoints of the animal object. That is, the object detection data212include a position (x1, y1) of the end of the head of the animal object, a position (x2, y2) of the neck and a position (xn, yn) of the end of the body.

The object detection data212may further include information about the class of the animal object detected from the images and information about a pose of the animal object. The information about the class of the animal object may distinguish different species of animals such as cows, pigs and dogs, but is not limited thereto. For example, the information about the class of the animal object may distinguish different growth stages of the same species. Pigs can be classified into suckling pigs, weaning pigs, growing pigs, fed pigs, candidate pigs, pregnant pigs and farrowing pigs. Also, the information about a pose of the animal object may distinguish various poses such as sitting, standing, mounting behavior, rollover, and dog sitting.

As shown inFIG.3A, the object images211may be composed of images cropped to sizes of respective bounding boxes created to be suitable for the animal object detected from the n number of continuous entire images200.

Examples of the object detection data212can be seen more clearly fromFIG.4. As described above, the bounding box of the present disclosure is created in consideration of the degree of rotation of the axis of the animal object as a detection target (i.e., an object), and, thus, the bounding box can be optimized for the size of the animal object.

Referring toFIG.3Aagain, the animal detection model600of the present disclosure is constructed based on the n number of continuous entire images200including at least one animal object and learning data in which animal condition information is matched with each class of each animal object included in each of the continuous entire images200. The animal detection model600is trained through a training process and then automatically extracts the animal detection information210including n number of continuous object images211from the n number of continuous entire images200and n number of continuous object detection data212corresponding to the respective object images211through an inference process on actually input images. A detailed configuration of the animal detection model600will be described later with reference toFIG.6throughFIG.8.

Hereafter, an animal condition identification model300that generates the animal condition information310will be described with reference toFIG.3BandFIG.3C.

Referring toFIG.3AandFIG.3B, the animal condition identification model300is constructed based on the n number of continuous entire images200including at least one animal object and learning data in which the animal condition information310is matched with each class of each animal object included in each of the continuous entire images200.

By way of example, the animal condition identification model300includes a first feature extraction unit301, a second feature extraction unit302and an output unit303.

The first feature extraction unit301generates n number of one-dimensional image data by converting the n number of continuous object images211into monochrome images and generates feature data of a first length based on the one-dimensional image data by using a convolutional neural network (CNN). For example, the first feature extraction unit301can be further improved in performance by using ResNet or DenseNet, which is a CNN classifier model improved over the CNN.

The second feature extraction unit302generates n number of one-dimensional data of a second length by connecting the n number of continuous object detection data212and generates feature data of the second length based on the one-dimensional image data of the second length by using a first feed-forward neural network (FFNN).

The output unit303generates data of a third length by connecting the feature data of the first length and the feature data of the second length and outputs the animal condition information310based on the data of the third length by using a second FFNN.

Also, the output unit303is constructed to sort an abnormal condition from the animal condition information310for each class of each animal object by using the softmax function.

Referring toFIG.3C, the animal condition information310is sorted by codes indicating abnormal conditions of animals including stop, walk, run, limp, delivery, excretion, heat, fell over, stuck, biting and nose rubbing.

For example, the second FFNN in the output unit303may finally generate, as an output value, the probability of each of the predefined animal condition information310as shown inFIG.3Cby using the softmax.

That is, an abnormal condition of a livestock animal is identified not with only one static datum as in the conventional technology, but with dynamic data, and, thus, it is possible to improve the identification accuracy. Also, various abnormal conditions of livestock animals ranging from short-term animal condition information such as walk, run and limp to long-term animal condition information such as delivery and disease can be identified depending on the number of continuous entire images.

Hereafter, the animal detection model600that generates the animal detection information210will be described.

FIG.6,FIG.7andFIG.8are provided to explain an animal detection model of the image-based animal object condition identification apparatus according to an embodiment of the present disclosure.

The animal detection model600includes a backbone610, a neck620and a head630.

The backbone610is a component configured to extract a feature from the input image and commonly used for deep neural network-based image analysis and processing. The backbone610is mainly configured as a stack of 2D convolution layers as illustrated inFIG.6, and has been improved to have various neural network structures in order to improve the efficiency thereof. Backbones of various structures commonly function to receive an image and extract intermediate information. The intermediate information is delivered to the neck620.

The neck620collects the intermediate information from each layer of the backbone610based on the feature extracted by the backbone610. The neck620is a lower neural network forming a universal object detector and functions to collect the intermediate information from each layer of the backbone610and analyze the intermediate information. The image analyzed in each layer has different resolutions. Thus, if an object is a long or short distance away, the neck620extracts intermediate information from each layer to effectively detect animals of various sizes and provides the intermediate information to the head630. The neck620may have various configurations depending on the form of the backbone610. Specifically, the number of layers of a neural network forming the neck620and a hyperparameter for each layer may vary depending on the form of the backbone610.

The head630outputs object detection information based on the intermediate information collected by the neck620. The head630receives the intermediate information acquired by the neck620and outputs animal detection information. The head630receives the intermediate information from each layer of the neck620and outputs the animal detection information recognized by each layer. In particular, the head630of the present disclosure includes a plurality of animal detection subnets631, and each animal detection subnet631includes a subnet for extracting a bounding box and a keypoint, a subnet for extracting a class of an animal and a subnet for extracting a pose of an animal as shown inFIG.7. That is, the animal detection model600may extract the n number of continuous object images211and the n number of continuous object detection data212corresponding to the respective object images211by means of each animal detection subnet631.

Meanwhile, a non-maximum suppression (NMS) module may be further coupled to an output end of the head630. The NMS refers to an algorithm for selecting a bounding box with the highest similarity when several bounding boxes are created for the same object. Since it is a conventional technology, a detailed description thereof will be omitted.

The subnet for extracting a bounding box and a keypoint is composed of cascaded multi-lane deep convolutional networks. The cascaded multi-lane deep convolutional networks are constructed according to a causal order for extracting a bounding box and a keypoint for a given animal image. Each of the object detection data212is defined from each image according to the following causal order.

That is, as shown inFIG.8, a central point (xc, yc) and major points ((x1, y1), (x2, y2), (x3, y3)) are marked first. Then, a tangent line passing through the central point and one or more of the major points is drawn. Finally, an area (plane) with the tangent line passing through its center is defined.

In the cascaded multi-lane deep convolutional networks constructed as described above, information is delivered according to the causal order and each information is output. That is, a first lane outputs the central point and the keypoint, a second lane outputs a direction (theta) of the tangent line, and a third lane outputs a width and a height of the area including the tangent line and the central point.

The learning data used in the training process of the animal detection model600include a plurality of images and the animal detection information210matched with each image. Herein, the animal detection information210is manually extracted from each image. That is, when an operator sees each image, the operator may use an appropriate SW tool to directly input the animal detection information210, or the animal detection information210may be automatically input by an already developed animal detector and then corrected or supplemented by the operator. For example, the operator displays a bounding box in consideration of a rotational direction of an animal object with respect to a reference axis for each animal object included in an image and creates coordinates of a central point of each bounding box, a width of the bounding box, a length of the bounding box and a rotational angle of the bounding box with respect to a reference axis. Also, the operator extracts information about the class or pose of the animal object and uses the information as learning data.

Hereafter, description of the same components as those shown inFIG.1throughFIG.8will be omitted.

FIG.9is a flowchart illustrating a method for identifying the condition of an animal object by using the image-based animal object condition identification apparatus according to an embodiment of the present disclosure.

Referring toFIG.9, the method for identifying the condition of an animal object by using the image-based animal object condition identification apparatus includes: a process of receiving the image10of an object (S110); a process of extracting the continuous animal detection information210of each object by inputting the received image into the animal detection model600that is trained based on learning data composed of animal images (S120); and a process of outputting the predetermined animal condition information310for each class of each animal object by inputting the continuous animal detection information210of each object into the animal condition identification model300(S130). The animal detection information210is extracted from the n number of continuous entire images200including at least one animal object, and includes the n number of continuous object images211and the n number of continuous object detection data212corresponding to the respective object images211.

The object detection data212refer to information about a bounding box created to be suitable for an animal object detected from each of the n number of continuous entire images200. That is, the object detection data212include coordinates of a central point of the bounding box, a width of the bounding box, a length of the bounding box and a rotational angle of the bounding box with respect to a reference axis. Also, the object detection data212refer to information indicating keypoints of the animal object. That is, the object detection data212include a position of the end of the head of the animal object, a position of the neck and a position of the end of the body.

The object images211are composed of images cropped to sizes of respective bounding boxes created to be suitable for the animal object detected from each of the n number of continuous entire images200.

The animal condition identification model300is constructed based on the n number of continuous entire images200including at least one animal object and learning data in which the animal condition information is matched with each class of each animal object included in each of the continuous entire images200.

The animal condition identification model300generates n number of one-dimensional image data by converting the n number of continuous object images211into monochrome images. The animal condition identification model300includes the first feature extraction unit301that generates feature data of a first length based on the one-dimensional image data by using a convolutional neural network (CNN). The animal condition identification model300includes the second feature extraction unit302that generates n number of one-dimensional data of a second length by connecting the n number of continuous object detection data212and generates feature data of the second length based on the one-dimensional image data of the second length by using a first feed-forward neural network (FFNN). The animal condition identification model300includes the output unit303that generates data of a third length by connecting the feature data of the first length and the feature data of the second length and outputs the animal condition information310based on the data of the third length by using a second FFNN.

The animal object condition identification method described above can be embodied in a storage medium including instruction codes executable by a computer such as a program module executed by the computer. A computer-readable medium can be any usable medium which can be accessed by the computer and includes all volatile/non-volatile and removable/non-removable media. Further, the computer-readable medium may include all computer storage media. The computer storage media include all volatile/non-volatile and removable/non-removable media embodied by a certain method or technology for storing information such as computer-readable instruction code, a data structure, a program module or other data.

It would be understood by a person with ordinary skill in the art that various changes and modifications may be made based on the above description without changing technical conception and essential features of the present disclosure. Thus, it is clear that the above-described embodiments are illustrative in all aspects and do not limit the present disclosure. The scope of the present disclosure is defined by the following claims. It shall be understood that all modifications and embodiments conceived from the meaning and scope of the claims and their equivalents are included in the scope of the present disclosure.

EXPLANATION OF REFERENCE NUMERALS

100: Image-based animal object condition identification apparatus110: Communication module120: Memory130: Processor140: Database