AGRICULTURAL FIELD ANALYSIS SYSTEM FOR GENERATING A FIELD DIGITAL TWIN

An aerial analysis system including an image gathering unit that gathers images of a field, an image analysis unit that analyzes images gathered by the image gathering unit, an information display unit that visualizes images identified in the gathered images, an information gathering unit that gathers information related to each gathered image, and a rule unit that generates a digital twin of the field based on the analyzed images and gathered information.

BACKGROUND OF THE INVENTION

The agriculture industry comprises a large portion of the world's economy. In addition, as the population of the world increases annually, more food must be produced by existing agricultural assets. In order to increase yields on existing plots of farmland, producers require a clear understanding of plant and soil conditions. However, as a single farm may encompass thousands of acres, it is difficult to access the conditions of the farmland.

Currently, farmers rely on their observations of their land along with prior experience to determine the requirements to increase the yield of their farmland. These observations may include identifying locations of weeds, identifying plant illnesses and determining levels of crop damage. However, considering the large number of acres in the average farm, these observations are not a reliable method to increase yields. Therefore, a need exists for system that will allow a farmer to better understand the conditions of their farmland.

SUMMARY OF THE INVENTION

One embodiment of the present disclosure includes an aerial analysis system that may have an image gathering unit that gathers images of a field, an image analysis unit that analyzes images gathered by the image gathering unit, an information display unit that visualizes images identified in the gathered images, an information gathering unit that gathers information related to each gathered image, and a rule unit that generates a digital twin of the field based on the analyzed images and gathered information.

In another embodiment, the information gathering unit gathers information on crops grown in the field.

In another embodiment, the information on crops includes soil information.

In another embodiment, the rule unit relates information from the gathered information to gathered images by applying at least one rule to each image.

In another embodiment, the rules applied by the rule unit are specific to each individual field.

In another embodiment, the rules applied by the rule unit are specific to a similar field.

In another embodiment, the information display unit modifies the image based on the rules applied.

In another embodiment, the image analysis unit modifies the image of the field by adjusting at least one piece of information related to the image.

In another embodiment, the image analysis unit determines the type of crop grown in the field.

In another embodiment, the image analysis unit determines the rows of crops from the image of the field.

Another embodiment of the current disclosure includes q method of generating a digital twin of a field, the method including gathering images of a field using an image gathering unit, analyzing the images of the field using an image analysis unit, visualizing the analyzed images using an information display unit, gathering information related to each image using an information gathering unit, and generating a digital twin of the field based on the analyzed images and gathered information.

In another embodiment, the information gathering unit gathers information on crops grown in the field.

In another embodiment, the information on crops includes soil information.

Another embodiment includes the step of relating information from the gathered information to gathered images by applying at least one rule to each image.

In another embodiment, the rules applied by the rule unit are specific to each individual field.

In another embodiment, the rules applied by the rule unit are specific to a similar field.

Another embodiment includes the step of modifying the image based on the rules applied.

In another embodiment, the image analysis unit modifies the image of the field by adjusting at least one piece of information related to the image.

In another embodiment, the image analysis unit determines the type of crop grown in the field.

In another embodiment, the image analysis unit determines the rows of crops from the image of the field.

DETAILED DESCRIPTION OF THE INVENTION

Referring now to the drawings which depict different embodiments consistent with the present invention, wherever possible, the same reference numbers will be used throughout the drawings and the following description to refer to the same or like parts.

The agricultural analysis system100gathers high to low resolution images from an aircraft flying above 1,500 feet. In one embodiment, the image is a multi-spectral image. The images are processed using various processing methodologies to determine a plurality of characteristics of a field including, but not limited to, the type of crop planted, number of rows, amount of weeds and any other characteristic of the field. Field observations, equipment operations and weather data is gathered and correlated with the field. By storing multiple images of fields over time, numerous profiles can be generated for each field. In addition, each field can be categorized and subcategorized to allow for correlations to be made between fields in different geographical locations and times.

FIG.1depicts one embodiment of a agricultural analysis system100consistent with the present invention. The agricultural analysis system100includes an agricultural analysis unit102, a communication device 1104, a communication device 2106each communicatively connected via a network108. The agricultural analysis unit102further includes an information gathering unit110, an information analysis unit112, a rule unit114and an information display unit116.

The information gathering unit110and information analysis unit112may be embodied by one or more servers. Alternatively, each of the rule unit114and information display unit116may be implemented using any combination of hardware and software, whether as incorporated in a single device or as a functionally distributed across multiple platforms and devices.

In one embodiment, the network108is a cellular network, a TCP/IP network, or any other suitable network topology. In another embodiment, the row identification device may be servers, workstations, network appliances or any other suitable data storage devices. In another embodiment, the communication devices104and106may be any combination of cellular phones, telephones, personal data assistants, or any other suitable communication devices. In one embodiment, the network102may be any private or public communication network known to one skilled in the art such as a local area network (“LAN”), wide area network (“WAN”), peer-to-peer network, cellular network or any suitable network, using standard communication protocols. The network108may include hardwired as well as wireless branches. The information gathering unit112may include a digital camera.

FIG.2depicts one embodiment of a agricultural analysis unit102. The agricultural analysis unit102includes a network I/O device204, a processor202, a display206and a secondary storage208running image storage unit210and a memory212running a graphical user interface214. The image gathering unit112, operating in memory208of the agricultural analysis unit102, is operatively configured to receive an image from the network I/O device204. In one embodiment, the processor202may be a central processing unit (“CPU”), an application specific integrated circuit (“ASIC”), a microprocessor or any other suitable processing device. The memory212may include a hard disk, random access memory, cache, removable media drive, mass storage or configuration suitable as storage for data, instructions, and information. In one embodiment, the memory208and processor202may be integrated. The memory may use any type of volatile or non-volatile storage techniques and mediums. The network I/O line204device may be a network interface card, a cellular interface card, a plain old telephone service (“POTS”) interface card, an ASCII interface card, or any other suitable network interface device. The rule unit114may be a compiled program running on a server, a process running on a microprocessor or any other suitable port control software.

FIG.3depicts one embodiment of a communication device104/106consistent with the present invention. The communication device104/1106includes a processor302, a network I/O Unit304, an image capture unit306, a secondary storage unit308including an image storage device310, and memory312running a graphical user interface314. In one embodiment, the processor302may be a central processing unit (“CPU”), an application specific integrated circuit (“ASIC”), a microprocessor or any other suitable processing device. The memory312may include a hard disk, random access memory, cache, removable media drive, mass storage or configuration suitable as storage for data, instructions, and information. In one embodiment, the memory312and processor302may be integrated. The memory may use any type of volatile or non-volatile storage techniques and mediums. The network I/O device304may be a network interface card, a plain old telephone service (“POTS”) interface card, an ASCII interface card, or any other suitable network interface device.

In one embodiment, the network108may be any private or public communication network known to one skilled in the art such as a Local Area

Network (“LAN”), Wide Area Network (“WAN”), Peer-to-Peer Network, Cellular network or any suitable network, using standard communication protocols. The network108may include hardwired as well as wireless branches.

FIG.4depicts a schematic representation of the components of the agricultural analysis system100. The agricultural analysis system100includes a central database402that collects information from a plurality of sources. Aerial imagery404is gathered from external sources including aircraft, drones, satellite images or any other aerial imagery source. Field observations406are gathered from individuals working in the fields. In one embodiment, the field observations are uploaded to the database402by workers working in a specific field. Field sensors408send information on soil conditions, water levels, environmental information and other measurable field information to the database402. Weather information410is gathered from third party sources. Weather information410may include, but is not limited to, rain levels, winds, sun exposure, humidity, temperatures and any other weather data. Equipment information is transmitted to the database402from equipment working a specific field. Equipment information may include, but is not limited to, equipment run time, speed, function, amount of discharge of product to the field and the location of the equipment in the field. In one embodiment, the equipment may include a camera that gathers high resolution images of the crops in the field. All information gathered by the database is related to the geographic location of the field.

The agricultural management unit102analyzes the information in the database to determine additional information for a field that is not directly measured. Using this information from the agricultural management unit102, the agricultural management unit102generates alerts that are transmitted to the party managing the field. In addition, the agricultural management unit102can make predictions on processes used in working the field and also in the overall yield of the field.

FIG.5depicts a schematic representation of a system of processing aerial images to extract information from images of fields. The aerial images500are gathered from aircraft or drones or satellite flying over the fields at an altitude of at least 1,500 feet above a field. The information analysis unit112applies a series of masks machine learning based models, computer vision algorithms and mathematical formulas to extract information form the aerial images and outputs the information502to the database402. The information502may include the health of the plants in the field, areas of active management, the soil conditions of the field, the arrangement of plants in the field, such as the arrangement, spacing and number of rows of plants in the field, the amount of weeds in the field and the amount of residual plant waste left in the field. In one embodiment, the information analysis unit112analyzed an image of a large area to determine the metes and bounds of parcels of land. In one embodiment, the information analysis unit112utilizes machine learning to determine the metes and bounds of a parcel. In another embodiment, information on the metes and bounds of a parcel are uploaded to the information analysis unit112. In another embodiment, the image analysis unit112gathers information from a Geographical Information System based on the location information associated with each image.

FIG.6depicts a schematic representation of a process used to correlate information related to a field. In step602, the geographic location of the field is identified. The geographic location may include, but is not limited to, the boundaries of the field, the GPS location of the boundaries of the field, the size of the field and any other geographic location indicators. In step604, the type of crop grown in the field is identified. The type of crop may be identified by analyzing the aerial image. In another embodiment, the type of crop is identified by a physical observation of the field. In another embodiment, the type of crop is identified through the uploading of equipment data. In step606, components of the field are extracted from the aerial images using the process described inFIG.5. In step608, any equipment used in the field that is transmitting information to the database402is added to the database. In step610, the sensors are associated with the field. In step612, field observations are associated with the field. In one embodiment, the identities of individuals are associated with the field.

FIG.7depicts a schematic representation of a process used to correlate equipment used in a field. In step702, an indication of operation is received from the equipment. The indication may be a wireless signal sent from the equipment to the agricultural analysis unit102. In step704, the type of operation is identified. The type of operation indicates what operation the equipment will be used for during the operation. In one embodiment, the type of operation of the equipment is transmitted from the equipment to the agricultural analysis unit102. In another embodiment, the type of operation of the equipment is transmitted to the agricultural analysis unit102by an observer. In step706, the start date and time of the operation is recorded. In step708, the information gathering unit112continuously gathers information from the equipment. The information gathering unit112may gather the speed the equipment is operating, discharge rates of any product discharged from the equipment, operational characteristics of the engine or any other information related to the equipment. In step710, the information gathering unit112tracks the path of the equipment through the field. In one embodiment, the equipment transmits the position of the equipment in the field to the information gathering unit112using GPS coordinates. In step712, the stop time of the operation of the equipment is recorded.

FIG.8depicts a schematic representation of a process used to verify or correct information gathered from aerial imagery. In step802, areas for confirmation are identified using geographic coordinates. The areas used for confirmation may include areas in the aerial imagery including information that was unknown and areas in the aerial imagery where the information is known and requires confirmation. In step804, a message is generated and transmitted to an individual in the field. In one embodiment, the individual receives the message on a communication device104/106when the individual is in proximity of the field.

In another embodiment, the information is preloaded onto a communication device before the individual enters the field. In step806, the information is transferred to the information gathering unit112and is used to update the characteristics of the field to enhance the accuracy of the analysis of the aerial imagery. In one embodiment, the user in the field gathers additional images using image gathering applications. As an illustrative example, a user may execute an application on a mobile communication device104/106that gathers and analyzes images taken by the user of specific objects, such as an ear of corn. In another embodiment, the application gathers location information of the user for later incorporation of the gathered data into the location associated with the aerial images.

FIG.9depicts a schematic representation of a process of categorizing fields. In step902, an aerial image of a field to be categorized. In step904, the type of crop being grown in the field is identified. In step906, the geographic location of the field is identified. In step908, the characteristics of the filed are identified. In step910, the information analysis unit112categorizes the field based on the type of crop grown on the field, the geographic location and the field characteristics. In step912, the information analysis unit112correlates relationships between the fields based on the crop grown, geographic location and the field characteristics.

FIG.10depicts a schematic representation of a process used to determine an unknown variable in a field. In step1002, an unknown variable of a field is identified. In step1004, the information associated with the field is retrieved along with the categorizations of the fields. In step1006, the information analysis unit112identifies similar fields. The similar fields may be any field in a same category of the field. In step1008, the information analysis unit112retrieves the information for each similar field. In step1010, the information analysis unit112compares the unknown information with known information in each similar field. In step1012, the information analysis unit112, normalizes the known values. In step1014, the information analysis unit112generates a value based on the normalized values.

FIG.11depicts a schematic representation of a process to apply a rule to the information related to a field. In step1102, a field to apply a rule to is identified. In step1104, a plurality of rules is retrieved based on the categories associated with the field. In step1106, the information related to the field is retrieved from the database402. In step1108, the rule analysis unit114analyzes the rule to determine the information and associated thresholds. In step1110, the rule analysis unit114determines field information includes all information and thresholds required to apply the rule to the field information. In step1112, the rule analysis unit114retrieves the next rule from the plurality of rules if the field information does not include the required information or the field information is outside the required threshold for the rule. In step1114, the rule analysis unit114applies the rule if all conditions of the rule are satisfied. In applying the rule, the rule may generate an alert to a user, modify the threshold of an associated rule, modify the display for a user or perform any other function.

FIG.12depicts a schematic representation of a process to generate a digital twin image of a field. In step1202, topographical information related to a field is retrieved. The topographical information may include field boundaries, elevations across the field, bodies of water, the grade of different areas in the field and any other topographical information related to the field. In step1204, crop information for the field is retrieved. The crop information may include, but is not limited to, the type of crop, the number and arrangement of rows, weed coverage, field anomalies, vegetation coverage, and any other information on the crop. In step1206, soil information is retrieved for the field. The soil information may include, but is not limited to, the soil moisture level, the compaction of the soil, the soil nutrient levels and any other soil information. In step1208, additional field information is gathered. The additional information may include, but is not limited to, locations of debris, locations of tress and other objects, amount of rain received by the field, amount of sun received by the field or any other information related to the field. In step1210, a representative image of the field is retrieved. In step1212, each piece of field information n is related to other pieces of field information by applying rules to the information that result in each piece of information being logically related to at least one other piece of information. In one embodiment, the rules are specific to the individual field. In another embodiment, the rules are related to similar fields. In step1214, the representative image is modified based on the field information and the correlations. In one embodiment, each piece of information gathered is converted into an object that is overlaid on the representative image. In another embodiment, the image is modified to reflect at least one piece of information. After all information is applied to the image, the image represents a digital representation of the field. By adjusting one or more variables, the digital representation of the field will react in a similar manner as the field in real life allowing for simulations [or other predictive models] to be run on the field representation to determine potential yield or other outcomes.

FIG.13depicts a schematic representation of a process of validating an alert. In step1302, a rule is applied to at least one piece of information related to a field or a piece of equipment related to the field. In step1304, predetermined thresholds for information associated with the rule is retrieved. In step1306, the predetermined thresholds are compared to corresponding pieces of information related to a field. If the corresponding values satisfy the predetermined threshold, an alert to a user in step1308. If the corresponding value does not satisfy the predetermined threshold, a new rule is selected in step1302. The user may receive the alert using any known method of transferring alerts including, but not limited to, emails, text messages, phone calls or any other method of transmitting an alert. In step1310, as part of the alert, the user is asked to acknowledge whether the alert provided useful information. In step1312, the alert and associated rule is left active if positive feedback is received from the user and the alert and associated rule are deactivated for the field if negative feedback is received.

FIG.14depicts a schematic representation of a process to simulate field conditions using the digital twin ofFIG.12. In step1402, at least one piece of information related to the field is adjusted to a new value. In step1404, information related to the field is retrieved. In step1406, historical information related to the field is retrieved. In step1408, the information analysis unit112determines if the new value of the adjusted information resides in any historical information. If the new value of the adjusted information is in the historical data, the field values are retrieved from the historical data and the field image is updated in step1414. In step1410, if the new value of the adjusted information does not reside in the historical data, the information analysis unit112analyzes the historical data to identify values similar to the adjusted value of the information. In step1412, the current field information values that were not adjusted are normalized based on the historical field information identified as having similar values to the adjusted information value. In step1414, the field image is modified based on the normalized values. In one embodiment, the field image is transmitted to an augmented/virtual reality engine that creates a augmented or virtual reality experience that displays the gathered information as part of the augmented or virtual reality interface. As one having ordinary skill in the art, will understand, by gathering information over a series of consecutive weeks, months and years, the accuracy of field calculations increases.

FIG.15depicts a schematic representation of a process to identify missing field information required to create a digital twin of a field. In step1502, current field information, including information with adjust values, are retrieved. In step1504, historical information for the field is retrieved. In step1506, similar fields are identified. In identifying similar fields, the crop type, geographic location, soil conditions and other field information is compared to information for a plurality of fields. The information analysis unit112filters field information to identify similar fields. In step1508, historical information is retrieved for the similar fields. In step1510, the current field information is compared to each similar field to identify similarities and differences in the information. In step1512, a normalizing factor is applied to the field information based on the similar field current and historical information. In step1514, the image is updated based on the normalized field information.

FIG.16depicts a schematic representation of a process used to identify objects in adjacent images. In step1602, a first image is gathered from memory or from an image gathering unit. In step1604, the image is correlated with the geographic location of the image. In one embodiment, the central portion of the image is used to define the geographic location. In another embodiment, the edges of the image are used to define the geographic location. In step1606, a second image is gathered. In step1608, the geographic location of the second image is correlated with the second image. In one embodiment, the central portion of the image is used to define the geographic location. In another embodiment, the edges of the image are used to define the geographic location. In step1610, the first image is correlated with the second image based on the geographic location of the first image. In one embodiment, the edges of the first and second images with similar geographical locations are logically related. In step1612, the first image is analyzed to identify objects in the image. In step1614, the second image is analyzed to identify objects in the second image. In step1616, the information analysis unit112correlates objects in the first image with objects in the second images based on the geographic location of the first image and second image.

FIG.17represents a schematic representation of a machine learning process used to analyze images. A machine learning unit1702includes a full supervised learning unit1704, a self-supervised learning unit1706and a semi supervised learning unit1708. The machine learning unit1702feeds data from the full supervised learning unit1704, self-supervised learning unit1704and semi supervised learning unit1708into a model analysis unit1710. The model analysis unit1710generates a model based on the received data and transfers the model to an inference unit1712. The inference unit1712analyzes data using the model from the model unit1710and transfers the analyzed data to an evaluation unit1714. The evaluation unit1714evaluates the model to determine the accuracy of the model. The inference unit1712also sends the analyzed data to a pseudo annotation unit1716. The pseudo annotation unit1716applies annotations to some of the data supplied from the inference unit1712and transfers the pseudo annotated data to the semi supervised learning unit1708. The pseudo annotation unit1716also transfers the pseudo annotated data to an annotation unit1718.

The annotation unit1718receives the pseudo annotated data and unannotated data from an unannotated data module1720. The annotation unit1718combines the pseudo annotated data with the unannotated data and annotates the unannotated data to generate a complete annotated data set. The annotation unit1718transmits the annotated data to an annotation module1722that feeds the annotated data to the full supervised learning unit1704and to the semi supervised learning unit1708. A synthetic data module1724also provides synthetic data to the full supervised learning module1704. The self learning unit1706receives information from the unannotated data module1722. The semi self learning unit1708receives data from the annotation module1722and the pseudo annotation unit1716.

While various embodiments of the present invention have been described, it will be apparent to those of skill in the art that many more embodiments and implementations are possible that are within the scope of this invention. Accordingly, the present invention is not to be restricted except in light of the attached claims and their equivalents.