Patent Publication Number: US-2019191632-A1

Title: Plant phenotyping techniques using optical measurements, and associated systems and methods

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     The present application is related to a U.S. application entitled “Improved Plant Phenotyping Techniques Using Mechanical Manipulation, and Associated Systems and Methods,” Attorney Docket Number XCOM164869, filed on the same day. 
     BACKGROUND 
     Plants are periodically evaluated in-field to estimate their size, stage of growth, sufficiency of watering, size of fruit, presence/absence of pests or disease, or other observable traits or characteristics. Such evaluation of plants is referred to as phenotyping. 
       FIG. 1A  is a picture of plants obtained in accordance with conventional technology. With some conventional technologies, the in-field phenotyping involves acquiring optical images of plants. These images are subsequently analyzed to establish relevant properties of the plants, for example, size of the plant, size of the fruit, etc. In many applications, the subsequent treatment of the plants (e.g., watering, application of pesticides, harvesting, etc.) is decided based on the analysis of the images. However, conventional imaging generates a large volume of relatively incomplete or difficult-to-analyze data. For example, parts of plants may be occluded or obscured such that relevant plant properties are difficult to derive from the images. Therefore, trained operators sometimes physically separate (physically “segment”) a plant  10  from the rest of the plants to make the outline of the plant  10  sharper and, thus, more suitable for subsequent analysis of the acquired image. However, such an individualized treatment of the target plant increases the cost and time of the in-field phenotyping. 
       FIG. 1B  is a picture of the plant  10  obtained in accordance with conventional technology. With the illustrated conventional technology, a physical backdrop  12  is placed behind the plant  10  to improve the isolation/contrast of the plant  10  against other plants in the field, therefore improving the sharpness of the image. As a result, the analysis of the image of the plant  10  is more accurate. However, placement of physical backdrops increases the time required for acquiring images and the cost of the phenotyping. 
       FIG. 1C  is a graph of plant phenotyping results obtained with conventional technology. With some conventional technologies, the internal or otherwise occluded plant features can be exposed by operators prior to imaging these plant features. For example, the operator may remove the husk that hides the corn ear structure prior to imaging corn kernels  14 . Once a relatively sharp outline of the corn kernels  14  is imaged, the size of the corn kernels may be obtained by fitting a suitable periodic curve  16  having amplitude and period that approximates the size of the corn kernels. Next, the curves  16  can be represented as a frequency-amplitude graph  17 . In the illustrated example, one or more peaks in the graph  17  correspond to the average length of the corn kernels  14 . However, this conventional technology results in a physical destruction of the corn ear structure that is evaluated. 
       FIG. 1D  is a picture of a field phenotyping system in accordance with conventional technology. An enclosure  25  carries a camera, while also limiting the amount of stray light around the plants. As a result, the images obtained by the camera are subjected to a more uniform intensity of light, which, in turn, makes subsequent analysis of the images more consistent. In operation, a tractor  20  pulls the enclosure  25  while the camera captures the images. However, this conventional technology requires additional equipment, namely, physical enclosures which must scale up in size as the plant grows, therefore increasing the cost of the phenotyping. 
     Accordingly, there remains a need for in-field plant phenotyping techniques and systems having a high-throughput and low cost of acquiring images that can be analyzed to produce accurate data about plants. 
    
    
     
       DESCRIPTION OF THE DRAWINGS 
       The foregoing aspects and many of the attendant advantages of the inventive technology will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein: 
         FIG. 1A  is a picture of a plant obtained in accordance with conventional technology; 
         FIG. 1B  is a picture of a plant obtained in accordance with conventional technology; 
         FIG. 1C  is a graph of plant phenotyping results obtained with conventional technology; 
         FIG. 1D  is a picture of a field phenotyping system in accordance with conventional technology; 
         FIG. 2  is a schematic view of a phenotyping system in accordance with embodiments of the present technology; 
         FIG. 3  is a schematic view of a phenotyping system in accordance with embodiments of the present technology; 
         FIG. 4  is a graph of the solar spectrum; 
         FIG. 5  is a schematic view of a phenotyping system operating in the H-α frequency band in accordance with embodiments of the present technology; 
         FIG. 6  is a schematic diagram of a trait extraction model in accordance with an embodiment of the present technology; 
         FIG. 7  is a schematic view of an analysis system in accordance with an embodiment of the present technology; 
         FIG. 8  is a flow diagram of a method for plant phenotyping in accordance with an embodiment of the present technology; and 
         FIGS. 9A and 9B  are graphs of plant detection in accordance with an embodiment of the present technology. 
     
    
    
     DETAILED DESCRIPTION 
     While illustrative embodiments have been described, it will be appreciated that various changes can be made therein without departing from the spirit and scope of the inventive technology. Embodiments of the inventive technology are generally applicable to the in-field, non-destructive phenotyping measurements of internal or occluded plant features, for example, size, stage of growth, sufficiency of watering, presence/absence of pests or disease, etc. 
     In some embodiments, multiple imaging modalities are used to acquire images of the plant(s) of interest to improve the accuracy and/or speed of the subsequent analysis of plant features (also referred to as “plant attributes”). Some examples of imaging modalities are camera sensors that operate within a frequency bandwidth (e.g., visible light, infrared light, X-rays, etc.), camera lenses that have different focal depths, optical or electromagnetic filters that transmit within a certain bandwidth while rejecting radiation outside of the bandwidth, and cameras that have different resolutions of the sensor. For example, one imaging modality may operate in the visible spectrum while another imagined modality operates in the X-ray spectrum. The imaging modality that operates in the visible spectrum may provide information about the size of the plant, color of the leaves, size of the fruit, etc., but the plant features that are occluded by proximate plants are not in the image, and, therefore, not available for analysis. On the other hand, the imaging modality that operates in the X-ray spectrum may not acquire very precise images about, for example, the outline of the plant, but the acquired images may reveal the occluded portions of the plant. In many embodiments, when the images obtained by different imaging modalities are analyzed as a group, relevant attributes of the plant can be defined more accurately. 
     In some embodiments, the system includes one or more dedicated sources that provide required electromagnetic or ultrasonic spectrum for the camera(s). For example, the system may include a source of ultrasound and an ultrasound camera (also referred to as a “receiver” or a “sensor” or “an ultrasound imaging modality”) for capturing the reflected ultrasound. Analogous pairs of source/imaging modality may operate in other spectra, for example, mm wave, X-ray, etc. In some embodiments, the imaging modalities, the sources, and/or analysis system may be carried by a ground vehicle or an air vehicle. In some embodiments, the vehicles may be unmanned. 
     In some embodiments, a source of mm-wave or microwave may heat the target plant. Generally, heating of the plant is a function of the properties of the plant, for example, size of the fruit, fraction of water in the plant, etc. Therefore, in some embodiments, the properties of the plant can be derived by analyzing images of the heated plants that were obtained in the infrared spectrum. 
       FIG. 2  is a schematic view of a phenotyping system  2000  in accordance with embodiments of the present technology. In some embodiments, the system  2000  includes several imaging modalities and sources that can operate either sequentially or simultaneously. The images acquired by the imaging modalities capture relevant features of the plant  40  (e.g., a stalk  42 , a husk  45 , a flower  46 , a corncob  48 , etc.). The images may be temporarily stored on the camera or forwarded to an analysis system for analyzing the images. The images can be indexed based on, for example, time of the image acquisition, properties of the imaging modality, for example, location, spectrum of operation, angle of view, direction, and other properties of the imaging modality. In some embodiments, the indexing may be based on determining that the first image and the second image include the same plant. 
     In some embodiments, the system  2000  may include a camera  120   a  configured to work in the visible spectrum. In some embodiments, the system  2000  includes pairs of sources and imaging modalities. For example, an X-ray source  120   b  may emit X-rays through the plant  40  toward an imaging modality  120   c  (e.g., an X-ray receiver or sensor). Furthermore, a source of ultrasound  120   e  may emit ultrasound that reflects off the plant  40  toward the imaging modality  120   d  (e.g., an ultrasound receiver or sensor). In at least some applications, the ultrasound penetrates internal features of the plant  40  before reflecting toward the imaging modality  120   d . Therefore, the images acquired by the imaging modality  120   d  may include plant features that are normally occluded, for example the features of the corncob hidden by the husk. Analogously, the images based on the X-rays or other electromagnetic spectra may capture features of the plant that are normally not available or not clearly outlined in the visible light spectrum. When analyzed as a group, the images acquired by different types of imaging modalities facilitate more precise and/or faster phenotyping of the plant. Such analysis of a group of images is sometimes referred to as sensor fusion for co-analyzing data from multiple sensors. 
       FIG. 3  is a schematic view of a phenotyping system  3000  in accordance with embodiments of the present technology. In some embodiments, the system includes a source of radiation (e.g., a source of electromagnetic waves)  120   b , and an imaging modality  120   c . For example, the source  120   b  may operate in the mm-wave or the microwave spectrum (i.e., the wavelength ranges from about 1 mm to about 30 cm), and the imaging modality  120   c  may operate in the infrared spectrum. In operation, the fruit  48  and the leaf  44  absorb the radiation from the source  120   b , resulting in a temperature increase. However, since the thermal mass of the fruit  48  is typically higher than that of the leaf  44 , the temperature rise of the fruit  48  and the leaf  44  above the ambient temperature are also typically different. Therefore, the images of these different parts of the plant are registered differently by the infrared imaging modality  120   c . Analogously, other parts of plants, for example, stalk, branches, flower, etc., that are illuminated by the source  120   b  also look different in the images acquired by the infrared imaging modality  120   c , depending on the thermal mass of the parts of plants. In some embodiments, based on the images acquired by the imaging modality  120   c , the analysis system can determine, for example, the mass, water content, size, ripeness, and other properties of the plant or parts of the plant. 
       FIG. 4  is a graph of the solar spectrum. The horizontal axis represents a subset of visible wavelengths from about 653 nm to about 660 nm. The vertical axis represents a normalized intensity of the solar light. The illustrated subset of the solar spectrum includes several dips in the light intensity. For example, for the wavelength of 656.28 nm (also referred to as the H-α wavelength or, in the context of frequencies, the H-α frequency), the normalized intensity of light drops to about 20% of the normalized light intensity in the shown range of the wavelengths. Other dips in the intensity of solar radiation exist. Some embodiments of the inventive technology that use the H-α wavelength or other relatively weakly-represented wavelengths are described with reference to  FIG. 5  below. 
       FIG. 5  is a schematic view of a phenotyping system  5000  operating in the H-α frequency band in accordance with embodiments of the present technology. In some embodiments, the phenotyping system  5000  includes an unmanned ground vehicle (UGV)  200  that carries the imaging modality  120  and a source of light  124  that emits light at the H-α wavelength. Some sources of light at the H-α wavelength are hydrogen discharge tubes, semiconductor lasers, light emitting diodes, and other sources. The imaging modality  120  may include a bandpass filter that is centered at around H-α wavelength to selectively allow H-α wavelengths toward the imaging modality. The illustrated system  5000  shows the UGV  200 , however, in other embodiments a manned vehicle or an unmanned aerial vehicle (UAV) may be used. The UGV  200  may carry a GPS (not shown). In some embodiments, the UGV may traverse the field with the plants  40  to obtain images that correspond to, for example, average properties of the plants in the field. 
     Since the solar emission at the H-α wavelength is generally weak, in many in-field situations the emission of the source  124  at the H-α wavelength is stronger than the H-α emission by the Sun. As a result, in some embodiments, the illustrated imaging modality  120  receives the H-α emission that is relatively independent or weakly dependent on the solar H-α emission. Therefore, the imaging modality  120  may acquire plant images of comparable intensity, for example, during day or night, in sunny or cloudy weather, etc. In some embodiments, a more uniform intensity of light results promotes a more accurate analysis of the plant attributes (properties). In some embodiments, the relative uniformity of the light generated by the source  124  provides a reference point that improves the calibration of the solar spectrum in the field. For example, the relatively small amount of transmitted solar H-alpha light may allow calibration of the current incident sunlight intensity, which is useful for understanding how much light the plants are receiving at the moment. 
       FIG. 6  is a schematic diagram of a trait extraction model in accordance with an embodiment of the present technology. In the illustrated embodiment, an analysis system  140  receives data  151  from one or more imaging modalities  151 . Furthermore, the analysis system may receive source data  152  from, for example, a source of ultrasound, a source of H-α light, a source of X-rays, etc. Some examples of the source data  152  are the intensity of the source, the angle of the source with respect to the plant of interest, the frequencies of emission of the source, etc. In some embodiments, the analysis system  140  receives ground truth data, for example, images of exposed corn ear, physical measurements of the stalk, observations about presence or absence of pests, etc. 
     The analysis system  140  includes software and instructions for analyzing images. In operation, the analysis system  140  processes the inputs using, for example, algorithms for digital image recognition. Based on the processing of the inputs  151 - 153 , the analysis system  140  evaluates plant properties  154 , for example, ripeness of the fruit, size and strength of stalk, water content of the leaves, etc. In some embodiments, the analysis system  140  is trainable to improve the evaluation of plant properties based on past analysis. 
       FIG. 7  is a schematic view of an analysis system in accordance with an embodiment of the present technology. In some embodiments, the analysis system  140  includes one or more processors  502  and a data storage  504 , such as a non-transitory computer readable medium. The data storage  504  may store program instructions  506 , which may be executable by the processor(s)  502 . The analysis system  140  may include the communication interface  121  for communication with the imaging modality. In different embodiments, the various components of the analysis system  140  may be arranged and connected in a different manner. 
       FIG. 8  is a flow diagram of a method  800  for plant phenotyping in accordance with an embodiment of the present technology. In some embodiments, the method may include additional steps or may be practiced without all steps illustrated in the flow chart. Furthermore, in some embodiments, the order of the steps may be changed. 
     The method starts in block  810 , and continues to block  820 . In block  820 , a target plant is identified for image acquisition. In some embodiments, a particular part of the plant, for example, the stalk or the fruit, is targeted for image acquisition. 
     In block  830 , the imaging modality acquires one or more images of the plant. The imaging modality may operate in a visible spectrum, in an X-ray spectrum, in the ultrasound spectrum, etc. In some embodiments, the system includes a source of ultrasound or electromagnetic radiation (e.g., X-rays, H-α spectrum, etc.). In some embodiments, a vehicle (e.g., UAV or UGV) carries the imaging modalities and the sources. 
     In block  840 , a decision is made whether to use an additional imaging modality. If the additional imaging modality is to be used, the method switches to that imaging modality in block  850 , and the additional images are acquired in block  830 . Switching between different imaging modalities may include adjusting camera settings, switching camera lenses, switching lens filters, switching illuminators, or switching between different physical camera types. If no additional imaging modality is to be used, the method proceeds to block  860  to analyze the images of plants. In some embodiments, the inputs to the analysis system include the ground truth data and/or the source data. 
     In block  870 , the analysis system determines the properties (attributes) of the plant. Some properties of the plant are size of the fruit, amount of water in the plant, etc. The method ends in block  880 . 
       FIGS. 9A and 9B  are graphs  900 A and  900 B of plant detection in accordance with an embodiment of the present technology. In both graphs, the horizontal axes correspond to time of frame acquisition. The vertical axes correspond to a distance that represents the presence of the target, which, in the illustrated case, is a strawberry fruit  48 . In some embodiments, a time series of images of the plant are acquired. The arrows point to the locations of the target fruit. In practical in-field situations, parts of the plant may be occluded by other parts of the same plant or by other plants, therefore impeding optical access to the object of interest. For example, the fruit  48  of the plant may be occluded by the leaves of the surrounding plants. The graph  900 A in  FIG. 9A  was obtained while the target fruit was occluded with one leaf, and the graph  900 B in  FIG. 9B  was obtained while the target fruit was occluded with six layers of leaves. In both cases, a radar signal is able to penetrate the leaf canopy and reflect a measurable signature of the strawberry fruit, therefore enabling identification of occluded fruit. 
     Many embodiments of the technology described above may take the form of computer-executable or controller-executable instructions, including routines stored on non-transitory memory and executed by a programmable computer or controller. Those skilled in the relevant art will appreciate that the technology can be practiced on computer/controller systems other than those shown and described above. The technology can be embodied in a special-purpose computer, application specific integrated circuit (ASIC), controller or data processor that is specifically programmed, configured or constructed to perform one or more of the computer-executable instructions described above. In many embodiments, any logic or algorithm described herein can be implemented in software or hardware, or a combination of software and hardware. 
     From the foregoing, it will be appreciated that specific embodiments of the technology have been described herein for purposes of illustration, but that various modifications may be made without deviating from the disclosure. Moreover, while various advantages and features associated with certain embodiments have been described above in the context of those embodiments, other embodiments may also exhibit such advantages and/or features, and not all embodiments need necessarily exhibit such advantages and/or features to fall within the scope of the technology. Accordingly, the disclosure can encompass other embodiments not expressly shown or described herein.