Automatic pest monitoring by cognitive image recognition with two cameras on autonomous vehicles

A computer-implemented method, an automatic pest control monitoring system and computer program product automatically monitor for pests on crops. An autonomous vehicle equipped with a normal camera and at least one alternate camera, simultaneously captures a normal image and a true alternate image containing a same portion of the crops. A composite image is generated using the difference of the captured images. If at least one pest is determined to present by applying an object recognition algorithm to the composite image, reactive measures are automatically deployed. The alternate camera may be an infrared camera or an ultraviolet camera. The composite image is generated by correlating the normal image with the true alternate image, determining the differences between the normal image and the true alternate image, and generating the composite image based on the differences between the normal image and the true alternate image.

BACKGROUND

The present disclosure generally relates to pest monitoring and control for crops and more particularly relates to a system and method for automatically monitoring pests using cognitive image recognition with different types of cameras on autonomous vehicles.

Pest control is an important task for agriculture to increase crop production. Monitoring plays a key role in pest control systems. Monitoring allows crop producers to identify the distribution of pests over their land and evaluate the impact of these pests on crop yield and quality. In addition, monitoring provides an ongoing pest history of the farms to improve farm management. Conventional monitoring mainly relies on traps (e.g., sticky trap, wing trap, bucket trap, pan trap, pitfall trap, light trap, etc.) or farmers sometimes capture pests themselves using vacuums or nets.

More recent methods of pest monitoring involve using drones to capture images of the plants and identify pests in the images. However, due to a variety of obstacles, such as the size and color of the plants and pests in relation to one another or the location of the pests on the plants (e.g., underside of leaves), pest identification and monitoring remains a constant challenge.

BRIEF SUMMARY

In various embodiments, a computer-implemented method, an automatic pest control monitoring system and computer program product automatically monitor for pests on crops are disclosed. The method comprises simultaneously capturing, by an autonomous vehicle equipped with a normal camera and at least one alternate camera, a normal image and a true alternate image containing a same portion of the crops. A composite image is generated using the captured images. If at least one pest is determined to present by applying an object recognition algorithm to the composite image, reactive measures are automatically deployed.

DETAILED DESCRIPTION

Embodiments of the present invention provide a system and associated methods to automatically monitor and control pests, such as insects or rodents, in agricultural/farming environments using an autonomous vehicle equipped with a Global Positioning System (GPS) receiver, a light and two different types of cameras. The vehicle can endure earth terrains and travels over a predefined track or route (e.g., corn rows in a field), simultaneously capturing photographs of the crops using both a normal camera and an alternate camera. Using the photographs captured by both cameras, a composite image is created which permits more accurate object recognition results to determine a more precise location and extent of a pest infestation quickly, allowing reactive measures to be taken at an earlier stage to contain and prevent widespread crop devastation. Reactive measures may include the autonomous vehicle immediately releasing traps and/or “on the spot” pesticides to mitigate damages caused by pests.

Operating Environment

Turning now toFIG. 1, a block diagram of one example of an operating environment comprising an automatic pest control monitoring system100using cognitive image recognition according to one embodiment of the present invention. The system100includes a pest control monitoring server102in wireless communication with at least one autonomous vehicle such as a remotely controlled flying drone104a, a land roving vehicle104b, and the like. Flying drone104aand land roving vehicle104bare referenced generally herein as “autonomous vehicle104.” AlthoughFIG. 1depicts one flying drone104aand one land roving vehicle104b, it should be noted that system100may contain any number of autonomous vehicles104in any configuration.

Pest control monitoring server102communicates wirelessly with autonomous vehicle104using any wireless communication protocol, such as cellular protocols (e.g., Long-Term Evolution (LTE), Global System for Mobile Communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), etc.), or short-range communication protocols such as that described by the IEEE 802.11 standard (i.e. Wi-Fi), Bluetooth, etc. It should be noted that the means of communication with the autonomous vehicle104is not limited to the above example communication protocols and may include any protocol that provides wireless communication. The pest control monitoring server102may provide instructions to direct the course, movement or speed of the autonomous vehicle102or the autonomous vehicle may be controlled via another wireless controller (not shown). Alternatively, the autonomous vehicle104may be pre-programmed to traverse a given path or route without receiving control feedback or input from the pest control monitoring server102.

In addition, the autonomous vehicle104may be in communication with a Global Positioning System (GPS) satellite106to determine the exact location (i.e. latitude and longitude) of the autonomous vehicle104at any given time. The autonomous vehicle104may send pest information (such as notification and identification of a pest outbreak, extent of infestation, number of pests identified, location of pest outbreak, actions deployed to combat outbreak, evidence of pest outbreak, etc.) to the pest control monitoring server102as the autonomous vehicle104traverses its route.

FIG. 2depicts a block diagram of an example autonomous vehicle104in accordance with the present invention. Autonomous vehicle104includes a processor202electrically coupled with a memory204and a communication interface206. The autonomous vehicle104communicates with pest control monitoring server102and any other remote controllers via the communication interface. Processor202is further electrically coupled to a variety of accessories including a locomotion means208, at least one alternate camera210, a normal camera212, a GPS transceiver214, a light216and a pesticide container218. The normal camera212captures images using the visible light frequency range. The alternate camera210may be an infrared camera, an ultraviolet camera, or some other type of camera different from a “normal” camera212, that captures images or visual representations of images using frequencies outside of the visible light spectrum. It should be noted that the accessories shown is not an exhaustive listing and may include other features.

The memory204contains program files and data files for use with the automatic pest control monitoring system100, such as driving module220, normal image222, simulated alternate image224, true alternate image226, composite image calculator228, composite image230, pest image recognition model232, object recognition algorithm234and pest log236. Normal images222are captured using a normal camera212and true alternate images224are captured correlating to the same or similar viewpoint as the normal image222using the alternate camera210. The selection of alternate camera210type may be dependent upon the type of pest and/or specific type of crop that the automatic pest control monitoring system100is observing. A simulated infrared alternate image226(e.g., a “fake” infrared image) is obtained, for example, by generating a greyscale image from the normal image222. Other known procedures for generating a simulated alternate image226from a normal image222may be used, but the details of the conversion process are outside the scope of this invention. A composite image calculator228creates a composite image230by finding the differences between the true alternate image224and the simulated alternate image226using methods which will be described in further detail below. An object recognition algorithm232uses the pest image recognition model234to determine whether any pests are located on or near the crops and the results are stored in pest log236.

AlthoughFIG. 2shows the simulated alternate image224, composite image calculator228, composite image230, pest image recognition model232, and object recognition algorithm234as being stored on the autonomous vehicle104, such that the processing of the images and object recognition is performed locally at the autonomous vehicle, some or all of these items could additionally or alternatively be stored on the pest control monitoring server102as the pest control monitoring server102may have greater computational and storage abilities. In that case, the autonomous vehicle104sends the normal image222and the true alternate image226to the pest control monitoring server102for analysis and the pest control monitoring server102returns results to the autonomous vehicle104for recording in the pest log236and taking any reactive measures.

The locomotion means208may be any means that allows for guided movement of the autonomous vehicle104. For example, in land roving vehicle104b, the locomotion means208may include an engine connected to a steering system and wheels for traversing over land. In flying drone104a, the locomotion means208may include an engine connected to a propeller and rudder system. The processor202controls the movement of the autonomous vehicle104using software stored in memory204as driving module220.

The light216is used to allow for navigation and/or normal image capturing in non-ideal conditions (e.g., nighttime, cloudy weather, etc.). Light may also be used to provide ultraviolet flash as an alternative to infrared.

When the automatic pest control monitoring system100positively identifies a pest infestation, the system100may deploy reactive measures to prevent crop damage, such as sending notifications of infestation to an authorized party, sounding alerts, automatically releasing pesticides, etc. Small quantities of pesticide may be carried in the pesticide container218for spot issues; however, larger pesticide distribution systems (not shown) covering most or all of the crop acreage may be connected to or in communication with the automatic pest control monitoring system100and may be activated upon a positive identification.

Exemplary steps for operating an automatic pest control monitoring system100using cognitive image recognition will be described with reference toFIG. 3.FIG. 3depicts an operational flowchart300for performing an exemplary process according to one embodiment of the present invention. At step302, the automatic pest control monitoring system100is trained to recognize known pests or evidence of pests (e.g., footprints, droppings, damage or breakage of plants, etc.). Using composite images230created from determining the differences between pairs of images consisting of a simultaneously captured true alternate image226and simulated alternate image224captured by the autonomous vehicle104or retrieved from a known database, a user with knowledge of known pests (e.g., farmer, entomologist, etc.) specifies the presence or absence of insects, other pests, or evidence of other pests by reviewing the composite images. The user may also identify (i.e. name) the pests during the training phase. It should be noted that even a trained human eye may have difficulty identifying some pests as the pests may tend to blend into the surrounding environment and be camouflaged to the naked eye. As a result of the training, a pest image recognition model is created for use during the monitoring phase of operation to identify pests.

At step304, the autonomous vehicle104monitors crops in the coverage area by following a predetermined route. As the autonomous vehicle104may be able to get lower to the ground than a person usually does, the vehicle104has a better view in monitoring such areas of the plants as the underside of leaves where insects may also live. At step306, the autonomous vehicle104simultaneously captures a pair of images: a normal image222from the normal camera212and a true alternate image226from the infrared camera210. Although pests are usually difficult to recognize using only a normal camera image, it is easier to identify the pests using an infrared camera because pests tend to have different temperature patterns than the surrounding environment.

The natural coloring of an insect often matches the surrounding leaves almost perfectly, providing camouflage to the insect and making detection difficult with the naked eye or in a normal color photograph. However, the contrast between the insect body and the surrounding environment is noticeably stronger in the true alternate image226captured using the alternate camera210, making identification a bit easier. In actual practice, the normal image222and the true alternate image226are correlated to line up such that the images substantially overlap perfectly. However, there is still a tendency for the insect body to blend in with the surrounding environment.

At step308, a composite image230is generated which emphasizes the insect body in relation to the surrounding environment (i.e. the contrast between the insect body and the surrounding environment is much greater). The composite image230is generated by first converting the normal image222which is captured in color, to a corresponding simulated alternate image224by converting the color pixels to greyscale or by other known methods. Each pixel of the greyscale image (i.e. simulated alternate image224) has a corresponding greyscale value. Likewise, each pixel of the true alternate image226also has a greyscale value. The composite image230is created by displaying the difference of the greyscale values of the corresponding pixels in both the simulated alternate image224and the true alternate image226on a pixel-by-pixel basis.

At step310, the trained pest image recognition model232is used with known object recognition techniques to determine the presence of pests. Optionally, the pest image recognition model232may be updated on the fly using results of the monitoring phase. If the automatic pest control monitoring system100determines that pests are present, at step312, reactive measures may be deployed at step314. As mentioned above, the reactive measures may include notifying an authority of the presence of pests, sounding an alert, or releasing pesticides, traps or other deterrents from the autonomous vehicle104. If no pests are found, at step312, the automatic pest control monitoring system100continues monitoring activities for a predetermined period of time, until the autonomous vehicle104reaches an endpoint of a predetermined route, or the system100is manually stopped.

Automatic Pest Monitoring Server

Referring now toFIG. 4, a block diagram illustrating an information processing system400that can be utilized in embodiments of the present invention is shown. The information processing system402is based upon a suitably configured processing system configured to implement one or more embodiments of the present disclosure (e.g., pest control monitoring server102). Any suitably configured processing system can be used as the information processing system402in embodiments of the present invention. The components of the information processing system402can include, but are not limited to, one or more processors or processing units404, a system memory406, and a bus408that couples various system components including the system memory406to the processor404.

Although not shown inFIG. 4, the main memory406may include normal image222, simulated alternate image224, true alternate image226, composite image calculator228, composite image230, pest image recognition model232and object recognition algorithm234shown inFIG. 2. One or more of these components can reside within the processor404, or be a separate hardware component. The system memory406can also include computer system readable media in the form of volatile memory, such as random access memory (RAM)410and/or cache memory412. The information processing system402can further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, a storage system414can be provided for reading from and writing to a non-removable or removable, non-volatile media such as one or more solid state disks and/or magnetic media (typically called a “hard drive”). A magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected to the bus408by one or more data media interfaces. The memory406can include at least one program product having a set of program modules that are configured to carry out the functions of an embodiment of the present disclosure.

Program/utility416, having a set of program modules418, may be stored in memory406by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. Program modules418generally carry out the functions and/or methodologies of embodiments of the present disclosure.

The information processing system402can also communicate with one or more external devices420(such as a keyboard, a pointing device, a display422, etc.); one or more devices that enable a user to interact with the information processing system402; and/or any devices (e.g., network card, modem, etc.) that enable computer system/server402to communicate with one or more other computing devices. Such communication can occur via I/O interfaces424. Still yet, the information processing system402can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and/or a public network (e.g., the Internet) via network adapter426. As depicted, the network adapter426communicates with the other components of information processing system402via the bus408. Other hardware and/or software components can also be used in conjunction with the information processing system702. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems.

As will be appreciated by one skilled in the art, aspects of the present disclosure may be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit”, “module”, or “system.”