Augmented reality for plant stand management

A plant stand management system includes a sensor unit configured to capture images of a plant stand, an applicator, and a controller communicatively coupled to the sensor unit and the applicator. The controller is configured to receive the captured images, process the captured images for determining one or more characteristics of the plant stand, generate one or more control signals based on the one or more characteristics, and send the one or more control signals to the applicator. The applicator is configured to perform at least one action on the plant stand based on the one or more control signals.

FIELD OF THE INVENTION

The present invention relates to plant stand management and, more particularly, to augmented reality for plant stand management.

BACKGROUND OF THE INVENTION

Late or out of place corn plants rarely produce an economical return in modern corn production. As a result, farmers invest heavily in planter technologies designed to increase the number of plants expected to produce a fully developed ear, as a percentage of the total plants that emerge. For example, if a farmer plants 34,000 seeds per acre, he expects (based on the pure live seed rating of his seed lot) at least 95% of the seeds to produce an emerging seedling. Until electronic innovations in planter row units became commercially available, data typically showed that perhaps only 80-90% of those emerging plants would produce a uniform and timely seedling under average conditions. Expressed as a percentage, this is referred to in the industry as the Net Effective Stand % (NES). Planter improvements have increased NES to perhaps 90% or more, removing approximately half of the non-effective plants. That still leaves approximately (conservatively) at least 5% of the plants as non-effective. In a stand of 34,000, this would represent 1700 or more plants that are in effect, weeds.

In other words, every 1% improvement in NES stands results in approximately 300 plants per acre, or over 2 bushels/acre, which on 80 million acres results in a gross benefit of 160 million bushels of corn, or nearly $0.5 B in revenue.

However, determining the location of such underperforming plants remains elusive, since they can be virtually anywhere within the stand. The task of finding them manually is costly and unrealistic. For example, evaluating plant stands such as corn have up until recently required manual counting and visual characterization by an agronomist or trained practitioner. In fact, such an evaluation is only conducted on an infrequent basis, and even then, only with small subsamples within a given field. More recently, unmanned aerial vehicles have developed camera technologies and applications to count individual plants, however, such technology has been unable to detect and/or quantify characteristics of individual plants. The remaining non-effective members of the population in the field therefore consume resources without the desired outcome.

What is needed in the art is a system and method for determining characteristics of individual plants, newly emergent and during the growing season, in an automated and efficient manner, then subsequently managing the plant stand, thereby optimizing the NES and harvest for the entire population.

SUMMARY OF THE INVENTION

In accordance with an aspect of the present invention, there is provided a system including an agricultural vehicle, and a plant stand management system mounted to the agricultural vehicle. The plant stand management system includes at least one sensor unit configured to capture images of a plant stand, at least one applicator, and a controller communicatively coupled to the at least one sensor unit and the at least one applicator. The controller is configured to receive the captured images, process the captured images for determining one or more characteristics of the plant stand, generate one or more control signals based on the one or more characteristics, and send the one or more control signals to the at least one applicator. Then at least one applicator is configured to perform at least one action on the plant stand based on the one or more control signals.

In accordance with another aspect of the present invention, a plant stand management system for use in an agricultural vehicle traversing a plant stand includes at least one sensor unit configured to capture images of the plant stand, at least one applicator, and a controller electrically coupled to the at least one sensor unit and the at least one applicator. The controller is configured to receive the captured images, process the captured images for determining one or more characteristics of the plant stand, generate one or more control signals based on the one or more characteristics, and send the one or more control signals to the at least one applicator. Then at least one applicator is configured to perform at least one action on the plant stand based on the one or more control signals.

In accordance with another aspect of the present invention, a method for plant stand management includes capturing, by at least one sensor unit, images of a plant stand, receiving, by a controller, the captured images, processing, by the controller, the captured images for determining one or more characteristics of the plant stand, generating, by the controller, one or more control signals based on the one or more characteristics, sending, by the controller, the one or more control signals to at least one applicator, and performing, by the at least one applicator, at least one action on the plant stand based on the one or more control signals.

DETAILED DESCRIPTION OF THE INVENTION

Referring now to the drawings, and more particularly toFIGS. 1-3, a system100formed in accordance with an embodiment of the present invention is shown. The system100includes an agricultural vehicle102including a chassis104having a front portion106and a back portion108. The vehicle102includes a cab110mounted to the chassis104, a storage container112mounted to the chassis104, and wheels114mounted to the chassis104. In the embodiment shown, the vehicle102also includes a boom116mounted to the back portion108of the chassis104, although the scope of the invention covers the boom116mounted to the front portion106of the chassis104. In one embodiment of the invention, the boom116is a conventional boom used in conjunction with applying agents, such as herbicides, pesticides, insecticides, fungicides and fertilizers, to a plant stand118. The agents may include plant and soil agents. The plant stand118may be a corn plant stand, however, the scope of the present invention covers all types of plants, and particularly, all types of crops, at all stages of growth.

By way of exemplary embodiments, the agricultural vehicle102may be a Case IH Patriot® sprayer or a New Holland Guardian® sprayer, however, the scope of the present invention covers all types of agricultural vehicles capable of being fitted for applying an agent to, or performing an action on, plants of a plant stand either individually or collectively, and/or to soil of a plant stand, including for example, planters, sprayers, harvesters, tractors and trucks, including pickup trucks.

The system100also includes a plant stand management system200mounted to the agricultural vehicle102. In one embodiment, the plant stand management system200includes at least one sensor unit202mounted to the boom116, at least one applicator204mounted to the boom116, at least one optional GPS device206mounted to the cab110, and a controller208mounted to the cab110, however, in other embodiments the sensor unit202may be mounted anywhere on the agricultural vehicle102and/or the GPS device206may be mounted anywhere on the agricultural vehicle102or on the boom116. The controller208is communicatively coupled to the sensor unit202, the applicator204, and the GPS device206via cables210,212,214, such as Ethernet coaxial cables forming a LAN, or wirelessly via a WLAN, for example.

An operator or an automated control system (not shown) of the vehicle102may adjust a height120of the boom116above a surface122of the plant stand118via, for example, a hydraulic system, based at least in part on a height124(or average height) of plants126of the plant stand118, conditions of the plant stand118or the surface122of the plant stand118, conditions of the soil of the plant stand118, settings of the sensor unit202and/or the applicator204, and/or on environmental conditions (e.g., wind, humidity and/or other ambient conditions).

In one embodiment of the invention, the sensor unit202is a camera configured to capture images of the plant stand118. By way of an exemplary embodiment, the camera may be a high-speed digital camera such as the EoSens® configured to capture 80-80,000 digital frames (i.e. digital images) per second with a maximum resolution of 1,280 by 1,024 pixels.

In one embodiment of the invention, each of the sensor units202are mounted to the boom116such that each sensor unit202is positioned above a corresponding row128of the plant stand118when the wheels114of the vehicle102are positioned between adjacent rows, e.g., rows128B,128C, for traveling parallel to the rows128. For example,FIG. 4Ashows an image402captured by the sensor unit202B positioned above the row128B, taken in a direction of the surface122of the plant stand118. As will be discussed in more detail further below,FIG. 4Bshows a processed image404, generated via application of image-processing software to the captured image402.

In another embodiment of the invention, each of the sensor units202are mounted to the boom116such that each sensor unit202is positioned between adjacent rows, e.g., rows128A and128B, of the plant stand118when the wheels114of the vehicle102are positioned between adjacent rows, e.g., rows128B and128C, for traveling parallel to the rows128. By way of an exemplary embodiment,FIG. 5shows a processed image500, generated from an image captured by the sensor unit202positioned between two adjacent rows, e.g., rows128A and128B, taken in a direction perpendicular to the rows128, showing processed images of plants126planted in rows128B and128C.

The applicator204may be mounted to the boom116such that each applicator204, e.g., applicator204A, is positioned above a row, e.g., row128A, for applying an agent to locations of targeted plants associated with the row128A, when the wheels114of the vehicle102are positioned between adjacent rows, e.g., rows128B and128C, for traveling parallel to the rows128. However, the scope of the invention covers the applicator204mounted between adjacent rows for applying the agent to locations of targeted plants belonging to at least one of the two adjacent rows. In one embodiment, the sensor units202are mounted to positions on the boom116that are forward (i.e., positioned closer to the front portion106of the vehicle102) of positions on the boom116to which the applicators204are mounted. By way of an exemplary embodiment, for vehicle102speeds of 10 mph, if the sensor units202are mounted to positions on the boom116that are approximately 18 inches forward of positions on the boom116to which the applicators204are mounted, and for plants126in a row128that are spaced 6 inches apart, the controller208will have up to a few tenths of a second to process the captured images and generate control signals for activating the applicators204. However, based upon the processing speed of the controller208and the speed of the vehicle, embodiments of the present invention include different distances in the forward direction between the sensor units202and the applicators204.

In an embodiment of the present invention, the controller306may include a processor216, e.g., a microcontroller or a CPU, and a memory218. The processor216is configured to receive the images, e.g., image402, captured by the one or more sensor units202, either directly from the sensor units202in real-time, or from the memory218or buffer (not shown) in which the captured images had been stored. The processor216then processes the received captured images for determining one or more characteristics of one or more plants126of the plant stand118. For example, in one embodiment of the invention, the processor216may execute image-processing code (i.e., software) stored in the memory218for determining one or more characteristics of one or more plants126of the plant stand118, or even of the plant stand118itself. In the following description, characteristics of the plant stand118includes characteristics of the plants126of the plant stand118.

By way of exemplary embodiments, the image processing code may include an image feature-detection code for processing the captured images, such as corner detection, blob detection, edge detection, or thresholding, and/or other known digital image feature detection means. If the captured image is not a digital image, the controller208may include means for digitizing the captured image. Such means are well known in the art.

The image processing code may also include image feature-position code for determining relative distances between features detected in the images (e.g., plant stems, leaves or other plant features), which may be further based on the settings (e.g., focal length, field of view) of the sensor unit202and/or height120of the sensor unit202above the surface122of the plant stand118. Some methods associated with the image feature-position code may assume objects with known geometry (or fiducial markers) are present in the image. However, if dimensions associated with features of portions of the image are unknown, simultaneous localization and mapping (SLAM) algorithms may be employed to map relative distances. Mathematical methods used in the image feature-position code may include projective (epipolar) geometry, geometric algebra, rotation representation with exponential map, Kalman and particle filters, nonlinear optimization and robust statistics, or other commonly known mathematical techniques.

The image processing code may also include image plant-location code for determining the global coordinates (e.g., a GPS location) of the plants126detected in the processed images based further on GPS signals received from the GPS device206and a relative location of the corresponding sensor unit202with respect to the GPS device206(i.e., the coordinates of the sensor unit202with respect to the GPS device206being located at an origin of a coordinate system).

In another embodiment of the present invention, the captured image (or otherwise digitized captured image) includes pixels, and the processor216executes, either as part of the image processing code or as a separate program, image identification code that determines, by way of an exemplary embodiment, RGB intensity values for each pixel of the captured image for determining whether the pixel is a pixel of a plant image (i.e., a plant pixel). The processor216then processes only the pixels identified as plant pixels, according to one or more components of the above-described image processing code, for determining one or more characteristics of each of the plants126captured in the images of the plant stand118.FIG. 4Bshows the image404, which is the result of the processor216processing the captured image402ofFIG. 4Aby executing one or more of the above-discussed codes of the image processing software.

In one embodiment of the present invention, the characteristics of a plant, as determined by the processor216, include a morphology value, a position of the plant126in relation to positions of other plants126in the plant stand118(e.g., distances between plants and/or distances of plants to other objects or features, such as plant rows128), and global coordinates (e.g., a GPS location) of the plant126in the plant stand118. The morphology value may be based on one or more of the following features of a plant, including but not limited to, plant stem size, plant height, number of leaves of the plant, dimensions of one or more of the leaves, and a quantization of the overall shape of the plant126. In one embodiment, a larger morphology value corresponds to a more mature plant and/or a healthier (i.e., more robust) plant.

In another embodiment of the invention, the characteristics of a plant stand, as determined by the processor216, may include an absence of a plant from a plant row128(i.e., a missing plant). For example, the image processing code may determine that one or more plants are absent from a row, based on an analysis of a current image(s) in combination with expected distances between plants in given plant row, either from data entered by an operator or on statistical data generated from previously processed images, for example.

The image processing code also may determine distances between plants126and between plants126and any other objects of the captured image, based further on one or parameters of the sensor unit202and/or one or more parameters of the plant stand118, including but not limited to, distances between the sensor unit202and the surface122of the plant stand118, a field of view of the sensor unit202or other optical characteristics of the sensor unit202, and dimensions of the plant stand (e.g., distances between adjacent rows of the plant stand118).

In another embodiment of the invention, the memory218is configured to store features of an average plant of the plant stand118. The features may be related to the morphology of an average or standard plant. An operator may enter features of an average plant based upon a visual survey of the plant stand118, or the processor216may be configured to generate features of an average plant of the plant stand118(e.g., a running average) based upon features determined from previously captured and processed images, and thereafter stored in the memory218. The processor216may then process a currently captured image for determining features of a given plant126, and compare the determined features with the features of the average plant stored in memory218to generate a morphology value of the given plant126.

For locating a plant within the plant stand118, the GPS device206provides GPS signals to the controller208. In addition, the memory218is configured to store relative locations of the corresponding sensor units202mounted to the agricultural vehicle102with respect to the GPS device206mounted to the agricultural vehicle102. The processor216processes the captured images from the sensor unit202, the GPS signals received from the GPS device206, and the locations of the sensor units202mounted to the agricultural vehicle102relative to the GPS device206mounted to the agricultural vehicle102for determining a GPS location of each plant captured in the image. In another embodiment of the invention, if the GPS device206is not able to acquire GPS signals over some intervals of time, then the processor216uses the speed of the vehicle102, acquired for example, from mechanical linkage and/or electrical circuitry coupled to the drive train (not shown) of the vehicle102, the last GPS location of the GPS device206computed from the last received GPS signal, which may be stored in memory218, the time lapse since the last received GPS signal, and the locations of the sensor units202mounted to the agricultural vehicle102relative to the GPS device206mounted to the agricultural vehicle102, with the current processed image (or set of current processed images) to determine the GPS location of the plants in the currently processed images.

In one embodiment of the preset invention, the GPS locations of the plants126of the captured images of the plant stand118are stored in the memory218, along with the other corresponding characteristics of each of the plants126, such as morphology values and distances with respect to other plants and/or other objects.

In another embodiment of the present invention, the processor216, after determining one or more characteristics of the plants126in the captured images, generates one or more control signals based on the determined characteristics and the locations of one or more applicators204mounted to the agricultural vehicle102relative to the GPS device206mounted to the agricultural vehicle102, which may be stored in and acquired from the memory218, and sends the control signals to the one or more of the applicators204for actuating one or more functions of the applicators204for performing an action on the plant stand118, including actions on targeted plants of the plant stand118.

According to one embodiment of the present invention, each applicator204includes at least one nozzle220coupled to at least one direction means222. For example, the direction means222may be in the form of electric, hydraulic, or pneumatic controlled actuators, including levers, pistons, pins or other known devices which convert a control signal to motion, connected to the nozzle220for controlling the direction of the nozzle220. The control signals may be based upon GPS locations of one or more targeted plants of the plant stand118, as well as the locations of the applicators204relative to the GPS device206, thereby enabling the direction means222to direct the nozzle220to point at a GPS location of a targeted plant by rotating and/or translating the nozzle.

In addition, each applicator204may be coupled to the storage unit112, containing the agent224, via at least one pump226and at least one actuator switch228, such as an electric, hydraulic, or pneumatic controlled fluid switch. AlthoughFIG. 3shows the pump226not integrated with the applicator204, the scope of the present invention covers a pump, e.g., pump226, integrated with one or more of the applicators204. In one embodiment, the actuator switch228receives one or more control signals for turning the switch off and on, thereby controlling a length of time that the switch remains open on (i.e., open) for controlling an amount of agent to be applied to a targeted plant location. In one embodiment, the storage unit112contains a plant agent, such as an herbicide directed to kill the targeted plants of the plant stand118, and the applicator204applies a determined dose of plant agent to the GPS locations of the targeted plants in the plant stand118, as directed by the one or more control signals, which are generated if the morphology value of the plant is below a threshold morphology value (e.g., if the plant is statistically small or not robust (i.e., not healthy) with respect to a standard representative plant or with respect to average features of those plants126of the plant stand118whose images had been previously captured and processed), or a distance between the location of the plant and a location of another plant of the plant stand is below a plant separation threshold value (e.g., the plants are growing too close together), or a shortest distance between the location of the plant and a nearest row is greater than a row offset value (e.g., the plant is growing too far from the row in which it was intended to be planted).

In one embodiment of the invention, the plant agent is an herbicide designated to kill targeted plants, such as, for example, plants of the plant stand118that are growing too close together, or growing too far away from a plant row, or are morphologically inferior (e.g., below a predefined morphology standard for plants as input by an operator, or below a standard deviation of morphology values of plants previously imaged in the plant stand118). However, the scope of the present invention covers all types of agents that have effects on plants and/or the plant stand, or on weeds of the plant stand, including but not limited to, pesticides, insecticides, fungicides, fertilizers, soil treatment agents, such as nitrogen, and even water.

In a further embodiment of the present invention, the processor216is configured to process the captured images for generating augmented reality (AR) images. Augmented reality is well-known to those of skill in the image-processing arts, the details of which will not be discussed here, except to mention that augmented reality includes processing techniques that augment real-world scenes by adding, subtracting, altering, and overlaying features to real-world images.FIG. 5shows as an AR image500, according to an embodiment of the present invention. The processor216may execute image processing code to augment the image of one or more plants of a group of plants that are growing too close to one another or are morphologically inferior and may augment the image of the plant stand118with symbols that represent missing plants (i.e., augment the image of target plants of the plant stand118and/or augment the image of the plant stand118itself). For example, when a distance between a location of a plant502and a location of a plant504is less than a predefined plant separation threshold value, the plants502and504, one or both of which may be referred to as a target plant, are augmented with a pre-defined color. In this embodiment, the image of the plants502and504are colored green and outlined in red, represented by the diagonal left-to-right striping. In addition, the processor216may augment the image of a plant when a shortest distance between the location of the plant and the nearest row is greater than a predefined row offset value. For example, as illustrated, a plant506is offset by a distance d508from the row128C (i.e., the shortest distance to the nearest row is the distance d to the row128C), and thus the image of the plant506is colored green and outlined in blue, represented by the diagonal right-to-left striping. The processor216may augment the image of a plant when the plant's morphology, as quantized by its morphology value, is less than a predefined morphology value threshold. For example, the image of the plant510is augmented orange with a blue outline, represented by the vertical striping, since it's morphological value, based upon one or more of height, number of leaves, leaf dimensions, or stem diameter, or even based upon a quantization of its overall shape (e.g., does it conform to a standard appearance of a corn plant at a given stage in its development) is below a predefined morphology threshold value.

Furthermore, the processor216may be configured to show missing plants associated with any given plant row128by adding augmentations to the image of the plant stand118that represent missing plants. In an exemplary example, the image500is augmented with plant images512at corresponding locations, having bodies colored green and outlined in white, represented by the horizontal striping, which represent plants that are expected to be at these locations in the plant rows128, based upon, for example, input from the user regarding crop planting statistics (e.g., seed separation, row spacings) and/or on average plant separation data gathered from previously processed images, but are absent (e.g., seeds were not planted at these locations, or seeds were planted, but did not germinate, or did germinate, but then succumbed to disease or environmental conditions). In addition, the processor216may execute the image processing code to augment the image of plants that are neither growing too close to one another nor are morphologically inferior, e.g., see augmented plant images514having bodies colored green and outlined in yellow (represented by the dotted pattern), thereby augmenting images of healthy, properly positioned plants.

Although the image500was augmented by colored images and/or symbols of the imaged plants, the scope of the invention covers any technique of differentiating and representing conditions of the plant stand118, such as augmenting the images of the plant stand118with 2D maps of colored polygons or pixels, or projecting 3D features onto the image of the 2D plant stand118.

In another embodiment of the present invention, the processor216may generate a first set of statistics associated with the plants126of the plant stand118based on captured and processed images. The first set of statistics may be based on the plant stand118as presently captured, before performing any action on the plant stand118and/or actions on targeted plants of the plant stand118by the applicator204(i.e., before managing the plant stand118via actions of the applicator204). The first set of statistics may include predicted plant growth rates, predicted soil nutrient levels and predicted plant stand (i.e., crop) yields based upon, for example, the number of plants detected in the plant stand, the morphology of the plants, the distances between plants and plant rows, and/or the global locations of the plants of the plant stand. In one embodiment of the present invention, the processor216generates the first set of statistics from the AR images.

The processor216may generate a second set of statistics associated with the plants126of the plant stand118based on the captured and processed images, and further based upon the expected changes to the plant stand118after performing, by the applicator204, actions to targeted plants and/or the plant stand118. For example, the second set of statistics may include predicted plant growth rates, predicted soil nutrient levels and predicted plant stand yields based upon application of the agent224by the applicator204to target plants, e.g., plants504,506,512, to eliminate plants growing too close together, or too far away from a designated row, or having undeveloped morphologies (low morphology values, for example). In one embodiment of the present invention, the processor modifies the AR images based upon the expected changes to the plant stand118after performing, by the applicator204, actions to targeted plants and/or the plant stand118, and then generates the second set of the statistics from the modified AR images. The controller208may be configured to store the first and second sets of statistics, as well as AR images, e.g., AR image500or spliced-together AR images depicting an agricultural field of which the plant stand118is a portion, for future display and/or further analysis.

In one embodiment of the present invention, the plant stand management system200comprises an optional display230(FIG. 3). The display230and the controller208may be integral components of a PC, a PDA, a laptop, or a smart phone, or the display230may be coupled to the controller208wirelessly or via cables232, such as Ethernet cables, and may be mounted to the vehicle102, for example to the cab110of the vehicle102.

FIG. 6is a flow chart illustrating a plant stand management method600, according to an embodiment of the present invention. In step605, images of a plant stand are captured. In one embodiment, one or more sensor units202captures images of the plants126of the plant stand118. The sensor unit202may be a high-speed digital camera. In one embodiment, one or more of the sensor units202may be mounted to an agricultural vehicle102, including mounting the sensor units202to an adjustable boom116of the vehicle102. As the vehicle102travels across the plant stand118, parallel to rows128of the plants126, the sensor unit202is configured to capture a series of images. Each sensor unit202may be configured to capture the images in a direction parallel to the rows128, in a direction perpendicular to the rows128or in any predefined angle with respect to a direction of the rows128. In one embodiment, each sensor unit202may be configured to simultaneously capture a series of pairs of images, each pair including one image taken in a first direction perpendicular to a row and a second image taken in a second direction 180° from the first direction.

In step610, the captured images are processed for determining one or more characteristics of the plant stand118, including one or more characteristics of one or more plants126of the plant stand118. The one or more characteristics of each plant126may include a morphology value, a position of the plant126relative to positions of other plants126in the plant stand118(e.g., distances between plants and/or distances of plants to other objects or features, such as plant rows, also referred to as relative distances of plants in the captured images), and a global location (e.g., GPS coordinates) of the plant126in the plant stand118. The morphology value may be based on at least one of the following features of a plant, such as plant stem size, plant height, number of leaves of the plant, dimensions of one or more of the leaves, and overall shape of the plant.

In one embodiment of the present invention, a processor216applies imaging processing software to the captured images, including image feature-detection code including corner detection, blob detection, edge detection, or thresholding, or other known image processing algorithms for determining features of plants detected in the captured images and computing a morphology value for each plant based upon one or more of its associated features, and/or image feature-position code for determining relative distances between features detected in the images (e.g., plant stems, leaves or other plant features), which may be further based on the settings and/or position of the sensor unit202relative to the surface122of the plant stand118, and/or plant-location code for determining the global coordinates (e.g., GPS location) of the plants126detected in the images based further on GPS signals received from the GPS device206and relative positions of the sensor unit202with respect to a position of the GPS device206.

In another embodiment of the present invention, the processor216applies RGB image identification software to the captured images for determining whether a pixel of the image belongs to a plant, based on its RGB value, and the processor216then applies one or more of the above-described image feature-detection code, image feature-position code and plant-location code only to those pixels of the captured image that are identified as plant pixels.

In step615, for those plants detected in the captured images having one or more determined characteristics meeting specifically predefined characteristics, where such detected plants are referred to as targeted plants, one or more associated control signals are generated. For example, one or more control signals associated with a plant (i.e. a targeted plant) may be generated when the morphology value of the plant is below a threshold morphology value, a distance between the plant and another plant is less than a plant distance threshold value, and/or a shortest distance between the plant and a nearest row is greater than a row threshold value.

In step620, one or more actions are performed on the targeted plants of the plant stand118based upon the control signals generated in association with the targeted plants. In one embodiment, one or more of the control signals generated in association with the targeted plant are received by an applicator204, and in an exemplary embodiment, by actuators222,228of the applicator204.

In one embodiment of the present invention, one or more of the applicators204may be mounted to an agricultural vehicle102, including mounting the applicators204to an adjustable boom116of the vehicle102. As the vehicle102travels across the plant stand118, parallel to rows128of plants126, the applicator204is configured to perform one or more actions on targeted plants of the plant stand118. In one embodiment, each of the applicators204and sensor units202are mounted to the boom116in such a manner that either each applicator204is above a row128when each sensor unit202is between adjacent rows, or each applicator204is above a row128when each sensor unit202is also above a row128, although the scope of the invention covers other mounting arrangements, based upon plant size and spacing between rows, for example. In one embodiment, each sensor unit202is mounted to the boom116in a position that is nearer the front portion106of the vehicle102than a position on the boom116to which each applicator204is mounted.

In an exemplary embodiment of the present invention, each applicator204comprises one or more nozzles220coupled to at least one directional means222, such as one or more directional actuators configured for receiving positioning control signals, such as electrical, pneumonic, or hydrolytic positioning control signals, that cause the nozzle220to rotate and/or translate for directing the agent224to a global location (e.g., a GPS location) of a targeted plant. Each applicator204may also include an actuator fluid switch or fluid valve228configured to receive one or more switching control signals for opening and/or closing the switch228, thereby coupling the nozzle220to storage container112containing the plant agent224via the switch228and a pump226.

In one embodiment of the present invention, when a plant is targeted because of distance to a neighboring plant, distance from its designated plant row, or morphology value, control signals are generated and received by the fluid valve228and positioning actuators222of the applicator204associated with the particular sensor unit202from which the targeted plant was imaged (e.g., the applicator mounted closest to the particular sensor unit) to cause the applicator204to apply, via a nozzle220, an herbicide to the GPS location of the targeted plant for a certain length of time in order to kill the plant. The length of time that the fluid valve228remains open to deliver a dose of the agent may depend upon the type of agent applied, the present environmental conditions (windy, wet, dry, cold, hot, etc.) and the size or morphological development of the plant.

In another embodiment of the present invention, the agent224is a fertilizer or water, and the targeted plants are malnourished or water-deprived plants. In yet another embodiment, all the plants126of the plant stand118are targeted, if, for example, each of these plants (or some predefined number of these plants) are underdeveloped, based upon the determined morphology values, for example, and thus a plurality of applicators204are activated to apply the appropriate agent224to the entire plant stand118.

In step625, the processed captured images are further processed for generating augmented reality (AR) images. For example, when a distance between a plant and another plant is below a predefined plant distance threshold value, the image processing software of the processor216is configured to apply an augmentation to the representation of one or both of the plants in the image, for example, by outlining the representations of the plants with a particular color, or by replacing the representation of one or both of the plants with a geometric figure or symbol that would represent plants growing too close together in the corresponding location of the plant stand118. Similarly, the image processing software of the processor216may apply other augmentations to representations of plants126in the image, or to the image of the plant stand118, when, for example, a shortest distance between a plant and a closest plant row is greater than a predetermined plant row distance threshold value, a morphology value of a plant is below a threshold morphology value, or locations in a plant row128at which a plant is to be expected, but is absent.

In one embodiment of the present invention, the processor216executes a two-step imaging process, generating first processed images based upon the captured images, as illustrated inFIG. 4B, from which the one or more characteristics of each plant126may be determined, and then generating second processed images (e.g., the AR images) based on the first processed images, which are particularly useful in presenting time-lapsed displays of the plant stand118, based upon images captured at different times during the growing season. In this manner, an operator may determine and illustrate (via the display230, for example) the effectiveness of applying, via the applicator204, different agents224on the development of the plant stand118and on the changes to the predicted yield as a result of applying the agent224as compared to the predicted yield if the agent224had not been applied.

In another embodiment of the present invention, the processor216executes a single-step imaging process, generating AR images based upon the captured images, as illustrated inFIG. 5, from which the one or more characteristics of each plant126may be determined and illustrated, via the display230, for example, for presenting time-lapsed statistics and/or maps of the plant stand118during the growing season, up until harvest.

FIG. 7illustrates a system700formed in accordance with another embodiment of the present invention. The system700includes an agricultural vehicle702. The reference numbers that are the same as the reference numbers of the previous figures designate identical features. The vehicle702includes a chassis104having a front portion106and a back portion108. The vehicle702includes a cab110mounted to the chassis104, a storage container112mounted to the chassis104, and wheels114mounted to the chassis104. In the embodiment shown, the vehicle102also includes a boom116mounted to the front portion106of the chassis104. In one embodiment of the invention, the boom116is a conventional boom used in conjunction with applying agents, such as herbicides, pesticides, insecticides, fungicides and fertilizers, to a plant stand118.

The system700also includes a plant stand management system200mounted to the agricultural vehicle702. The plant stand management system200includes at least one sensor unit202mounted to the boom116, at least one applicator204mounted to the boom116, at least one optional GPS device206mounted to the cab110, and a controller208mounted to the cab110. The controller208is communicatively coupled to the sensor unit202, the applicator204, and the GPS device206via cables210,212,214, such as Ethernet coaxial cables forming a LAN, or wirelessly via a WLAN, for example.

As shown, the sensor units202are mounted forward of the applicators204. The sensor units202may be mounted to the boom116via attachment means203, such as arms. The attachments means203are rotatably attached to the boom116, thereby allowing the sensor units202and the attachment means203to be rotated into a position parallel to, or in plane with, a longitudinal axis of the boom116to facilitate storage of the sensor units202, particularly when not in use, and/or to facilitate the boom116to be folded to facilitate storage of the vehicle702, transport of the vehicle702, and maneuverability of the vehicle702.FIG. 8illustrates the vehicle702with the boom116folded. For example, in one embodiment of the present invention, each half of the boom116may be folded at two points117and119, although the scope of the invention covers all configurations of foldable booms. For ease of illustration, the sensor units202, the attachment means203and the applicators204are not shown inFIG. 8.

It is to be understood that the steps of the method600are performed by their respective controller208upon loading and executing software code or instructions which are tangibly stored on a tangible computer readable medium, such as on a magnetic medium, e.g., a computer hard drive, an optical medium, e.g., an optical disc, solid-state memory, e.g., flash memory, or other storage media known in the art. Thus, any of the functionality performed by the controller208described herein, such as the method600, is implemented in software code or instructions which are tangibly stored on a tangible computer readable medium. Upon loading and executing such software code or instructions by the controller208, the controller208may perform any of the functionality of the controller208described herein, including any steps of the method600described herein.