SYSTEMS AND METHODS FOR AUTONOMOUS MONITORING AND/OR OPTIMIZATION OF PLANT GROWTH

A system for autonomous monitoring and/or optimization of plant (140) growth is provided. The system may include actuation devices configured to interact with an agricultural area (120), image sensors (130) configured to capture images (170) of a plant (140) in the agricultural area (120), and a processor (150) in communication with the image sensors (130) and the actuation devices. The processor (150) may be configured to store, via a memory (160), a first image (170) of the agricultural area (120) captured prior to a first actuation of the actuation devices; and trigger, synchronously with the first actuation, the image sensors (130) to capture a second image (180) of the agricultural area (120). The processor (150) may be further configured to detect features of the plant (140) in the first and second images (170) of the agricultural area (120); evaluate the detected features of the plant (140) for visual plant qualities (210); and dynamically set one or more parameters (190) of the actuation devices based on the visual plant qualities (210).

FIELD OF THE DISCLOSURE

The present disclosure is directed generally to systems and methods for autonomous monitoring and/or optimization of plant growth.

BACKGROUND

The world population is expected to reach 9.8 billion by 2050, 68% of which will be living in urban areas. While the need for food is on the rise, agricultural land area dwindles due to urbanization and climate change. As a result, most of the urban population has little or no access to fresh and healthy produce nearby. Indoor farming is quickly becoming a preferred and efficient way to produce more food with fewer resources than conventional farming, without being dependent on arable land availability and external climate conditions.

According to a 2017 market report, labor accounts for 56% of operational expenses in vertical farms. On average, a vertical farm requires 63 laborers per acre, whereas greenhouses need only 3. In vertical farms, when not carefully screened upon every entry, laborers may be carriers of contamination. Accordingly, a greater degree of automation in the vertical farming industry is needed to reduce labor cost and risk of contamination.

SUMMARY OF THE DISCLOSURE

The present disclosure is directed to inventive systems and methods for performing autonomous monitoring and/or optimization of plant growth, for example, by analyzing images and other data captured in an agricultural area before and after an actuation of an actuation device. Based on the analysis, the actuation device is dynamically set to provide conditions to improve plant growth. In some examples, the actuation devices may be set to provide mist via an irrigation system, light via a lighting system, and/or air flow via e.g. a fan. Further, if the system is unable to capture the plant or specific features of the plant in an image, the actuation devices may be dynamically set to aid in capturing the images. In one example, the lighting provided by the lighting system may be adjusted such that the plant is more visible in subsequently captured images.

The invention is defined by the independent claims and the corresponding depending claims.

Generally, in one aspect, a system for autonomous monitoring and/or optimization of plant growth is provided. The system may include one or more actuation devices configured to interact with an agricultural area. The system may further include one or more image sensors configured to capture images of a plant in the agricultural area.

The system may further include a processor in communication with the one or more image sensors. The processor may also be in communication with the one or more actuation devices. The processor may be configured to store, via a memory, a first image of the agricultural area captured prior to a first actuation of the one or more actuation devices by the one or more image sensors. The processor may be further configured to trigger, within a predetermined period following the first actuation, the one or more image sensors to capture a second image of the agricultural area. The processor may be further configured to detect one or more features of the plant in the first and second images of the agricultural area. The processor may be further configured to evaluate the detected one or more features of the plant for one or more visual plant qualities.

The term “synchronously with the first actuation”, as used in the present application, may mean within a predetermined period following the first actuation, wherein the predetermined period may be e.g. at most 1 minute, at most 30 seconds, or at most 1 second.

According to an example, the processor may be further configured to determine if the plant is present in the first and second images of the agricultural area. The processor may be further configured to dynamically set one or more parameters of the one or more actuation devices if the plant is not present in the first and second images.

According to an example, the processor may be further configured to determine if one or more features of the plant are visible in the first and second images of the agricultural area. The processor may be further configured to dynamically set one or more parameters of the one or more actuation devices if the one or more features of plant are not present in the first and second images.

According to an example, as partly mentioned, the processor may be further configured to detect one or more features of the plant in the first and second images of the agricultural area. The processor may be further configured to evaluate the detected one or more features of the plant for one or more visual plant qualities. The processor may be further configured to dynamically set one or more parameters of the one or more actuation devices based on the one or more visual plant qualities.

According to an example, the one or more visual plant qualities may not be visible on the detected one or more features of the plant in the first image.

As it is already mentioned that if the system is unable to capture the plant or specific features of the plant in an image, the actuation device may be dynamically set to aid in capturing the images. This may similarly apply if the system is unable to capture the plant or specific features of the plant in an image, the first actuation of the actuation device may aid in capturing the images, and in particular the one or more visual plant qualities on the detected one or more features of the plant in the first image. The first actuation may in aspects be unrelated or independent to the timing of the one or more image sensors, e.g. a first actuation may occur independently anyhow, e.g. a scheduled misting. For example, in a non-limiting example: a fungus on a leave may not be visible on the first image, but the processor may trigger the one or more image sensors within a predetermined period following said first actuation to capture a second image, which first actuation may e.g. be a scheduled misting independent from the image capturing of the one or more image sensors, so as to aid the determining of the fungus on the leave in the second image. Since the mist may deflect the leave, and therefore the fungus may become more visible. Similar examples may be envisioned.

According to an example, the first actuation may be scheduled. According to an example, the first actuation may be independent from the one or more image sensors capturing the first image, and the processor storing the first image via the memory.

According to an example, the system may further include one or more environmental sensors configured to capture environmental data in the agricultural area. The environmental sensors may include one or more electrochemical soil monitoring sensors. The processor may be further configured to store, via the memory, a first environmental dataset captured in the agricultural area prior to the first actuation of the one or more actuation devices. The processor may be further configured to trigger, synchronously with the first actuation, the one or more environmental sensors to capture a second environmental dataset in the agricultural area. The processor may be further configured to evaluate the first and second environmental datasets for one or more environmental qualities. The dynamic setting of the one or more parameters may be further based on the one or more environmental qualities.

According to an example, the processor may be further configured to identify the location of the one or more features of the plant in the first and second images. The evaluation of the one or more features of the plant may be further based on the location of the one or more features of the plant.

According to an example, the processor may be further configured to trigger the one or more image sensors to capture the second image of the agricultural area within a predetermined (or scheduled) period following the first actuation.

According to an example, the image sensors may include one or more red-green-blue (RGB) sensors. The image sensors may include one or more multi-spectral sensors.

According to an example, the actuation devices may include one or more lighting systems. The actuation devices may include one or more irrigation systems.

Generally, in another aspect a method for autonomous monitoring and/or optimization of plant growth is provided. The method may include retrieving, from a memory, via a processor, a first image of the agricultural area captured prior to a first actuation of one or more actuation devices. The method may include triggering, via the processor, synchronously with the first actuation, the one or more image sensors to capture a second image of the agricultural area. The method may include detecting, via the processor, one or more features of a plant in the first and second images of the agricultural area. The method may include evaluating, via the processor, the detected one or more features of the plant for one or more visual plant qualities.

According to an example, the method may further include dynamically setting one or more parameters of the one or more actuation devices based on the one or more visual plant qualities.

According to an example, the method may further include retrieving, from the memory, a first environmental dataset captured in the agricultural area prior to the first actuation of the one or more actuation devices. The method may further include triggering, via the processor, synchronously with the first actuation, the one or more environmental sensors to capture a second environmental dataset in the agricultural area. The method may further include evaluating the first and second environmental datasets for one or more environmental qualities. The method may further include dynamically setting one or more parameters of the one or more actuation devices based on the one or more environmental qualities and/or the one or more visual plant qualities.

These and other aspects of the various embodiments will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.

DETAILED DESCRIPTION OF EMBODIMENTS

The present disclosure is directed to inventive systems and methods for performing autonomous monitoring and/or optimization of plant growth More generally, Applicant has recognized and appreciated that it would be beneficial to provide a system configured to monitor the growth and health of plants within an agricultural area, and to subsequently adjust the conditions of the agricultural area in response to the status of the plants. Applicant has also recognized and appreciated that it would be beneficial to create such a system which adjusts for difficulties in monitoring the plants, such as when portions of a plant are occluded in a captured image to be analyzed.

Referring toFIGS.1and2, in one aspect, a system100for autonomous monitoring and/or optimization of plant growth is provided. The system100may include one or more actuation devices110configured to interact with an agricultural area120. The actuation devices110may include one or more lighting systems. The lighting system may be configured to control the color, color temperature, intensity, direction, modulation, light scene, light recipe, and/or lighting cycle of one or more luminaires within the agricultural area. The actuation devices110may include one or more irrigation systems. The irrigation system may be configured to provide water to one or more portions of the agricultural area120. The water may be provided in the form of a mist. The actuation devices110may be a temperature control system configured set the temperature of one or more portions of the agricultural area120. The actuation devices110may be one or more water spray guns. The actuation devices110may be one or more robotic arms configured to move any component of the system, such as the image sensors130. The actuation devices110may be one or more robotic arms configured to interact with the agricultural area120, including the plants140within the agricultural area120. The actuation devices110may be one or more vibrating units. The actuation devices110may be one or more audio speakers. The actuation devices110may be one or more fans or blowers configured to circulate air through one or more portions of the agricultural area120. The actuation devices110may be any combination of the aforementioned systems and devices.

The system100may further include one or more image sensors130configured to capture images of a plant140in the agricultural area120. According to an example, the image sensors130may include one or more red-green-blue (RGB) sensors. The image sensors130may include one or more multi-spectral sensors. The image sensors130may include one or more time of flight sensors. The image sensors130may include any combination of the aforementioned systems and devices. The image sensors130may be mounted to mechanical arms, robotic arms, drones, or any other device capable of positioning the image sensor130within the agricultural area120. Examples of images captured by image sensors130are shown inFIGS.3-5.

The system100may further include a processor150in communication with the one or more image sensors130. The processor150may also be in communication with the one or more actuation devices110. The processor150may be configured to store, via a memory160, a first image170of the agricultural area120captured prior to a first actuation of the one or more actuation devices110. Processor150may be capable of executing instructions stored in memory160or otherwise processing data to, for example, perform one or more steps of a method. Processor150may be formed of one or multiple modules. Processor150may take any suitable form, including but not limited to a microprocessor, microcontroller, multiple microcontrollers, circuitry, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), a single processor, or plural processors.

Memory160may be a non-transitory medium. Memory160can take any suitable form, including a non-volatile memory and/or RAM. The memory160may include various memories such as, for example L1, L2, or L3 cache or system memory. As such, the memory160may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read only memory (ROM), or other similar memory devices.

The processor150may be further configured to trigger, synchronously with the first actuation, the one or more image sensors130to capture a second image180of the agricultural area120. As described in greater detail below, the processor150may then subsequently compare the second image180to the first image170to evaluate the plant140.

According to an example, the processor150may be further configured to trigger the one or more image sensors130to capture the second image180of the agricultural area120within a predetermined (or scheduled) period270following the first actuation. The predetermined (or scheduled) period270may be set automatically through the optimization process. The predetermined (or scheduled) period270may be stored in memory160. The predetermined (or scheduled) period270may be programmed by a user via a user interface. The user interface may include one or more devices for enabling communication with a user. The user interface can be any device or system that allows information to be conveyed and/or received, and may include a display, a mouse, and/or a keyboard for receiving user commands. In some embodiments, the user interface may include a command line interface or graphical user interface or micro services that may be presented to a remote terminal via communication interface. The user interface may be located with one or more other components of the system, or may located remote from the system and in communication via a wired and/or wireless communications network.

The predetermined period270may depend on the type of analysis. For example, if the system100is determining whether one or more features200of plant140are visible in first and second images170,180, the predetermined period270of time may be a period of seconds, such as 10 seconds. In a further example, if the system100is evaluating the growth of plant140, the predetermined period of time may be a period of hours, such as 10 hours. Further, the predetermined period270may depend on the variety of plants140grown in the agricultural area120. Additionally, the predetermined period270may depend on the type and characteristics of the image sensors130and/or actuation devices110used by the system100. The predetermined period of time270may be any length of time appropriate for the present analysis.

According to an example, the processor150may be further configured to determine if the plant140is present in the first and second images170,180of the agricultural area120.FIG.2shows a number of plants140within agricultural area120to be detected by the processor150. The determination of the presence of the plant140may be executed using a convolutional neural network implementing a binary classification scheme. Broadly, the binary classification scheme classifies portions (such as individual pixels or groupings of pixels) of the images170,180as containing a plant or not. The neural network may be characterized by a sequence of blocks each consisting of convolutional layers, max pooling layers, and an activation layer. During a training process, the network learns the optimal classification parameters using a large training set of images with and without plants. Users may also supply manually annotated labels for each image as ground truth data.

The processor150may be further configured to dynamically set one or more parameters190of the one or more actuation devices110if the plant140is not present in the first and second images170,180. For example, the parameters190of a lighting system may be set to increase brightness in poorly lit portions of the agricultural area120.

In a further example, the processor150may be configured to determine if the plant140is present in the first image170of the agricultural area120, rather than both the first and second images170,180. The first image170is captured prior to the first actuation of the one or more actuation devices110. The processor150may then be further configured to dynamically set one or more parameters190of the one or more actuation devices110if the plant140is not present in the first image170. In this configuration, the processor150may dynamically set the parameters190without requiring the first actuation.

Additionally, in determining the presence of a plant140in an agricultural area120, the processor150may be configured to forego the first actuation. In this example, the processor150may determine the presence of the plant140in the first image170and second image180without an intervening first actuation in between the capture of the two images170,180. The processor150may then be further configured to dynamically set one or more parameters190of the one or more actuation devices110if the plant140is not present in the first and/or second images170,180.

According to an example, the processor150may be further configured to determine if one or more features200of the plant140are visible in the first and second images170,180of the agricultural area120. The features200of the plant140may be, for example, flowers, leaves, nuts, and/or roots. The feature200may not be present in the images170,180due to occlusion by other plants140or other features200of the same plant140.FIG.3shows an example of such occlusion, wherein a flower is occluded by several leaves.

The processor150may be further configured to dynamically set one or more parameters190of the one or more actuation devices110if the one or more features200of plant140are not present in the first and second images170,180. For example, the parameters190of a robotic arm may be set to move the leaves of plants140to remove an occlusion of one or more flowers. In another example, the parameters190of a fan or blower may be set generate an air flow to move the leaves of plants140to remove an occlusion of one or more flowers. In a further example, the parameters190of a water spray gun may be set generate a water spray to move the leaves of plants140to remove an occlusion of one or more flowers.

In a further example, the processor150may be configured to determine if one or more features200of the plant140are visible in the first image170of the agricultural area120, rather than both the first and second images170,180. The first image170is captured prior to the first actuation of the one or more actuation devices110. The processor150may then be further configured to dynamically set one or more parameters190of the one or more actuation devices110if the one or more features200of the plant140are not present in the first image170. In this configuration, the processor150may dynamically set the parameters190without requiring the first actuation.

Additionally, in determining the visibility of plant features200in the agricultural area120, the processor150may be configured to forego the first actuation. In this example, the processor150may determine the visibility of the features200of the plant140in the first image170and second image180without an intervening first actuation in between the capture of the two images170,180. The processor150may then be further configured to dynamically set one or more parameters190of the one or more actuation devices110if the features200are not visible in the first and/or second images170,180.

In aspects, the processor may be further configured to dynamically set one or more parameters of the one or more actuation devices if the one or more features of plant are not present in the first image. For example, the parameters of a robotic arm may be set to move the leaves of plants to remove an occlusion of one or more flowers. In another example, the parameters of a fan or blower may be set generate an air flow to move the leaves of plants to remove an occlusion of one or more flowers. In a further example, the parameters of a water spray gun may be set generate a water spray to move the leaves of plants to remove an occlusion of one or more flowers. This beneficially creates a system which adjusts for difficulties in monitoring the plants, such as when portions of a plant are occluded in a captured image to be analyzed.

According to an example, the processor150may be further configured to detect one or more features200of the plant140in the first and second images170,180of the agricultural area120. The features200, such as flowers, may be identified using a semantic segmentation algorithm. The algorithm may be configured distinguish portions (such as individual pixels or groupings of pixels) of the images as specific plant features200and all other objects. In an example, an auto-encoder model based on U-Net architecture may be used. An example of such detection is shown inFIG.4. InFIG.4, the upper image of the agricultural area120is processed such that the flowers within the image are replaced with white pixels, and all other objects within the image are replaced with black pixels.

The processor150may be further configured to evaluate the detected one or more features200of the plant140for one or more visual plant qualities210. The visual plant qualities210may include the amount, size, and/or color of the features200of the plant140. For example,FIG.4may be evaluated to determine that fifteen (15) flowers exist within the image. Further, the white areas of the lower image ofFIG.4may be analyzed to determine the size of the flowers within the image. In a further example,FIG.5may be evaluated to determine a discoloration in the flower. The visual plant qualities210may include relative measurements, such as the rate of growth of the features200in between the capturing of the first image170and second image180.

According to an example, the processor150may be further configured to identify the location260of the one or more features of the plant140in the first and second images170,180. The evaluation of the one or more features200of the plant140may be further based on the location260of the one or more features200of the plant140. For example, the processor150may quantify the change in location (or movement) of the leaves of the plant140when an irrigation system applies a mist or when a fan applied an air flow.

The processor150may be further configured to dynamically set one or more parameters190of the one or more actuation devices110based on the one or more visual plant qualities210. For example, the discoloration shown inFIG.5may cause the parameters190of an irrigation system to mist the plant140with a greater amount of water than previously provided. In a further example, the discoloration shown inFIG.5may cause the parameters190of a lighting system to provide the plant140with a greater amount of sunlight than previously provided. The parameters190may be set by a comparison of the visual plant qualities210to desired values, such as size, color, or growth rate. These desired values may be programmed using a user interface and stored in the memory160.

According to an example, the system100may further include one or more environmental sensors220configured to capture environmental data in the agricultural area120. The environmental sensors220may include one or more carbon dioxide sensors. The environmental sensors220may include one or more volatile organic compound (VOC) sensors. The environmental sensors220may include one or more temperature sensors. The environmental sensors220may include one or more humidity sensors. The environmental sensors220may include one or more microphones. The environmental sensors220may include one or more sensors for monitoring the quality of milk of animals fed with the plants140. The environmental sensors220may include one or more electrochemical soil monitoring sensors. The electrochemical soil monitoring sensors may be configured to measure pH level, electrical conductivity, electromagnetic properties, evapotranspiration rate, moisture content, and/or nitrogen content. The electrochemical soil monitoring sensors may be placed at multiple different depths. The environmental sensors220may include any combination of the aforementioned systems and devices.

The system100may further include a user interface electrically coupled to the processor150wherein a user may enter environmental data. The user interface may be further configured for the user to enter data related to the schedule and/or availability of workers or consumers in the agricultural area. For example, if the system100requires or prefers human intervention, knowledge of a holiday would inform the system that workers are unavailable, and to proceed with alternative optimization options (such as automated lighting or irrigation adjustments).

The processor150may be further configured to store, via the memory160, a first environmental dataset230captured in the agricultural area120prior to the first actuation of the one or more actuation devices110. The first environmental dataset230may be captured at any appropriate point within the agricultural area120. For example, the first environmental dataset may be captured proximate to one or more plants140.

The processor150may be further configured to trigger, synchronously with the first actuation, the one or more environmental sensors220to capture a second environmental dataset240in the agricultural area120. According to an example, the processor150may be further configured to trigger the one or more environmental sensors220to capture the second environmental dataset120of the agricultural area120within a predetermined period270following the first actuation.

The processor150may be further configured to evaluate the first and second environmental datasets230,240for one or more environmental qualities250. The environmental qualities250may include temperature, soil pH, carbon dioxide level, or any other quality of the agricultural probative as status of the agricultural area120. The environmental qualities250may be evaluated in a similar manner as the visual plant qualities210.

The dynamic setting of the one or more parameters190may be further based on the one or more environmental qualities250. For example, the intensity of the luminaries of the lighting system may be adjusted according to the temperature of the agricultural area120. In a further example, a robotic arm may deposit fertilizer in portions of the agricultural area120with low nutrient measurements.

Referring toFIG.6, in another aspect of the present invention, a method400for autonomous monitoring and/or optimization of plant growth is provided. The method400may include retrieving410, from a memory, via a processor, a first image of the agricultural area captured prior to a first actuation of one or more actuation devices. The method400may include triggering420, via the processor, synchronously with the first actuation, the one or more image sensors to capture a second image of the agricultural area. The method400may include detecting430, via the processor, one or more features of a plant in the first and second images of the agricultural area. The method400may include evaluating440, via the processor, the detected one or more features of the plant for one or more visual plant qualities.

According to an example, the method400may further include dynamically setting450one or more parameters of the one or more actuation devices based on the one or more visual plant qualities.

According to an example, and with reference toFIG.7, the method400may further include retrieving460, from the memory, a first environmental dataset captured in the agricultural area prior to the first actuation of the one or more actuation devices. The method400may further include triggering470, via the processor, synchronously with the first actuation, the one or more environmental sensors to capture a second environmental dataset in the agricultural area. The method400may further include evaluating480the first and second environmental datasets for one or more environmental qualities. In this example, the dynamic setting of the one or more parameters of the one or more actuation devices may be further based on the one or more environmental qualities.