Patent Description:
Although satellite imaging can be helpful for predicting activities of natural systems and how they may affect certain agricultural operations, satellite imaging may lack precise data, e.g., at the individual plant level, which otherwise could be harnessed to increase agricultural yields. It is possible to traverse various types of robots through agricultural areas to capture data at the individual plant level, e.g., using two-dimensional ("2D") and/or three-dimensional ("3D") sensors. However, many robots (also referred as "rovers") perform localization using position coordinate sensors, such as global position system ("GPS") sensors, that are often not sufficiently accurate or consistent to localize the robots at an individual plant level. Moreover, GPS may not operate in some scenarios, such as when crops are planted underneath or near a natural or man-made canopy.

<CIT> discloses a system, method, and apparatus for automatically gathering seed-specific data for an agricultural crop. Simulated seeds with contactless machine-readable data are co-mingled with actual seeds. Whether in stored form prior to planting, during planting, or after planting with the actual seed in the ground, appropriate readers can quickly and accurately read the seed-specific data for a variety of purposes. That can include simply confirming that the actual seed at least in proximity to a simulated seed is of a particular hybrid or variety. It could also include other seed-specific data such as time and date of planning, seed production company, seed-specific usage restrictions, etc. The data can be utilized by other systems. One example would be a precision agricultural system.

Implementations set forth herein relate to incorporating fiducial markings onto surfaces of agricultural apparatuses, such as synthetic mulch, in furtherance of collecting and analyzing agricultural data at a level of granularity that is difficult to achieve using conventional position coordinate sensors. The agricultural data can be generated by a mobile computing device (e.g. a robot or rover) that is tasked with identifying environmental conditions affecting agricultural areas at which the fiducial markings are located. Thereafter, the data can be used for assisting other autonomous agricultural devices that operate to inspect plants, perform routine agricultural tasks, and/or diagnose issues that may affect an environment of certain plants. In some implementations, the fiducial markings can be located on synthetic mulch, ground cover, drip tapes, hoses, tubing, emitters, pipes, dirt, mulch, plants, and/or any other feature of land that can assist with providing some amount of agricultural product.

In some implementations, the fiducial markings used to gather and/or track data can be located on synthetic mulch, which can be used for a variety of purposes such as maintaining soil moisture and/or preventing growth of weeds. A fiducial marking can be a label (e.g., a biodegradable sticker) that is applied to the synthetic mulch, or can be directly printed on the synthetic mulch. The fiducial marking can be recognizable by the mobile computing device and/or any suitable autonomous agricultural device. In any implementation discussed herein, an RFID, a Bluetooth beacon, and/or other apparatus that is detectable via electromagnetic signal, can be used in place of each fiducial marking and/or in combination with each fiducial marking. In this way, the apparatus can be detectable even when produce or other material may be covering the apparatus. The mobile computing device can scan the fiducial marking using one or more sensors of the mobile computing device and correlate the fiducial marking to plant data that is stored in association with a plant that may be most proximate to the fiducial marking. Features of the plant and/or environmental conditions of the plant can then be determined and embodied as plant-specific agricultural data that can be stored in association with the fiducial marking.

As an example, synthetic mulch can be disposed over an agricultural area in which a plurality of plants are, or will be, growing. The synthetic mulch can include an array of fiducial markings. The fiducial marking may be regularly spaced so that when plants are planted through the synthetic mulch, one or more fiducial markings are likely to be proximate each plant, or an algorithm can interpolate the distance between fiducials to determine the location of each plant. Before, during, and/or after a plant has yielded some amount of produce, the plant can be monitored by an autonomous agricultural device. The autonomous agricultural device can identify each fiducial marking, access agricultural data that is stored in association with the fiducial marking, and perform one or more maintenance operations on plants based on the agricultural data.

For instance, prior to planting or germination of a plurality of plants, a mobile computing device can navigate through an agricultural area, over which synthetic mulch is disposed, in order to identify each fiducial marking on the synthetic mulch. Each time the mobile computing device detects a fiducial marking, the mobile computing device can use one or more sensors to identify features of an environment surrounding the fiducial marking and then generate data based on the features of the environment. Thereafter, the data can be stored in association with the fiducial marking. For example, in some implementations, GPS data can be captured and stored in association with each fiducial marking (or a subset of markings), in order that autonomous agricultural devices may not have to rely on satellite GPS data in order to subsequently locate each plant. Rather, the autonomous agricultural devices can rely on the previously stored data and, optionally, compare the data to a known location in order to identify a location of a plant.

In some implementations, an ability of a robot to read every fiducial marking may not be necessary in order to determine and navigation to a location of a particular plant. Rather, the robot can interpolate locations when some markings are missing, obstructed, and/or damaged based on each fiducial marking corresponding to a pattern-based sequence and/or other predictable sequence. Additionally, or alternatively, when a fiducial marking is generated and/or a seed is initially planted, environmental conditions of the seed, the soil, the synthetic mulch, the weather, and/or any other information can be captured and stored in association with the fiducial marking. Such initial data can later be referenced by (i) the autonomous agricultural devices when performing subsequent maintenance near the plant and/or (ii) other systems and/or devices that can use the initial data for furthering other agricultural processes.

In some implementations, during plant growth, an autonomous agricultural device can be tasked with performing routine maintenance in order to keep each plant healthy during a growing and harvesting period. Such time can be crucial to farmers, as much produce can be susceptible to destruction by pests, over or under watering, malnourishment by deprivation of certain nutrients by weeds or other organisms, growth abnormalities, and/or any other features or qualities that can be experienced by a plant during growth. Moreover, during the growth of produce from a plurality of plants, the produce can exhibit positive qualities that may be indicated by their size, color, shape, texture, temperature, scent, weight, and/or any other quality that can be exhibited by a plant. Therefore, any of the positive and/or negative features experienced during plant growth can be detected by the mobile computing device and/or autonomous agricultural devices, and used to generate agricultural data that can be correlated with each plant.

As an example, agricultural data for a plant can be generated whenever a pest is detected near the plant in order that a problem and/or solution for the pest problem can be identified and used to prevent or resolve other pest issues in other agricultural areas. For instance, trends in pest populations can be identified based on multiple data entries in order to predict areas that may be subsequently affected by pests. In this way, in order to prevent future plant destruction by pests, certain deterrents can be employed in order to mitigate opportunities for pests to damage produce from plants predicted to be targeted by pests. Moreover, as such data is tracked over time for various agricultural areas and/or various different plants, an efficacy of certain pest deterrents can be measured. For example, when an autonomous agricultural device is tasked with assisting farmers with deploying pest deterrents, the autonomous agricultural device can identify each plant by a nearby fiducial marking (e.g., a fiducial marking disposed over a drip tube), identify a particular deterrent (e.g., an organic chemical deterrent) that is suitable for the type of plant, and deploy the deterrent as the autonomous agricultural device traverses an agricultural area.

When plants in an agricultural area are producing fruits, vegetables, and other produce, various data associated with such produce can be generated and stored in association with each respective fiducial marking. For example, an autonomous agricultural device that collects produce from a plant can identify a fiducial marking corresponding to the plant and generate production data characterizing a current total yield of produce from that particular plant. The production data can be used to identify correlations between the production data and various environmental changes that may have affected the particular plant over time.

In some implementations, a combination of fiducial markings can embody a wide-area fiducial marking that can be visible from above the agricultural area. For example, an array of fiducial markings can be visible from a vehicle that is flying over the array of fiducial markings, and the array of fiducial markings can be unique relative to other arrays of fiducial markings. A plurality of plants disposed about that area that includes an array of fiducial markings can be affected by wide-area environmental changes, which can also be visible from certain elevations and detected by other devices occupying such elevations. Such wide-area environmental changes can be characterized by agricultural data that can be stored in association with each fiducial marking that is part of the array of fiducial markings. For example, data characterizing an environmental condition can be stored in association with an identifier for a particular array of fiducial markings. Furthermore, that data can also be stored in association with each identifier for each fiducial marking of the particular array of fiducial markings. In this way, similar environmental changes that are predicted to affect other agricultural areas can be identified and used to mitigate any negative effects of the environmental changes to those other agricultural areas.

In some implementations, data that is collected and stored in association with the fiducial markings can be used to track produce that may eventually exhibit positive and/or negative qualities after the produce is moved outside of the agricultural area for consumption. As one non-limiting example, annual crops such as tomatoes can be subsequently used in the manufacturing of products such as tomato sauce, ketchup, etc. Prior to such manufacturing, the tomatoes can be analyzed to determine their quality, health, and/or other properties. When a batch of tomatoes are inspected at a manufacturing facility and determined to exhibit a deficiency (e.g., as indicated by abnormal texture and/or discoloration), a computer system at the manufacturing facility can generate data characterizing the deficient features of the batch of tomatoes. The manufacturing facility can transmit the data to an entity that manages the agricultural data corresponding to the batch of tomatoes and the fiducial markings located at the agricultural area from which the tomatoes arrived. The data can then be stored in association with one or more identifiers corresponding to one or more fiducial markings located proximate to particular tomato plants that yielded the particular batch of tomatoes analyzed at the manufacturing facility. In this way, any autonomous agricultural devices subsequently traversing the agricultural area that includes those particular tomato plants can identify the particular tomato plants via their assigned fiducial markings and perform some amount of maintenance (e.g., remove or add material or organism) to remedy deficient conditions of the particular tomato plants.

As used herein, when a plant is said to have yielded "deficient produce," that may refer to the plant having generated produce that exhibits deficiencies such as those described above in relation to tomatoes, or it may refer to the plant underperforming such that its produce yield fails to meet an expected yield, or both.

Other implementations may include a non-transitory computer readable storage medium storing instructions executable by one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and/or tensor processing unit(s) (TPU(s)) to perform a method such as one or more of the methods described above and/or elsewhere herein. Yet other implementations may include a system of one or more computers that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and/or elsewhere herein.

<FIG> illustrates a view <NUM> of a vehicle <NUM> that arranges synthetic mulch <NUM>, having fiducial markings <NUM>, over an agricultural area <NUM>. The synthetic mulch can be pre-printed with the fiducial markings <NUM>, and each fiducial markings <NUM> can be generated to convey a unique symbol or other 1D, 2D, and/or 3D marking. Alternatively, or additionally, each fiducial marking <NUM> can be printed onto the synthetic mulch <NUM> by/at the vehicle <NUM>, simultaneous to the vehicle <NUM> laying the synthetic mulch <NUM> onto the agricultural area <NUM>. Alternatively, or additionally, each fiducial marking <NUM> can be an apparatus that includes an RFID tag and/or one or more sensors for detecting properties of the agricultural area <NUM> such as, but not limited to, moisture, soil pH and/or soil content, temperature, living organisms, and/or any other features or properties that can be exhibited by soil.

As provided in <FIG>, the fiducial markings <NUM> can include indicia that is detectable by one or more computing devices, such that a particular computing device can correlate the indicia to: stored data, one or more plants in the agricultural area <NUM>, equipment that operates in the agricultural area <NUM>, produce from the one or more plants in the agricultural area <NUM>, and/or any other feature or information that can be associated with an agricultural area. For instance, the fiducial markings <NUM> can include a printed symbol, such as at least printed symbols 106A, 106B, 106C, and 106D.

In some implementations, each printed symbol can be a unique, sequential identifier, but can be arranged in a similar orientation relative to other printed symbols. In some implementations, each printed symbol can include a one-dimensional barcode, a two-dimensional or matrix bar code such as a Quick Response® ("QR") code, a sequences of characters/numbers/symbols, and/or any other marking that can be uniquely generated by a computing device. Alternatively, or additionally, one or more printed symbols can be the same as one or more other printed symbols on the synthetic mulch <NUM>, and the one or more printed symbols can have the same or a different orientation than the one or more other printed symbols on the synthetic mulch <NUM>.

In some implementations, each additional fiducial marking <NUM> can be arranged to indicate a direction in which the finished marking <NUM> is oriented. As an example, the printed symbol 106A can be rotated <NUM> degrees counter-clockwise from the printed symbol 106C, a printed symbol 106C can be rotated <NUM> degrees counterclockwise from the printed symbol 106B, and a printed symbol 106D can be rotated <NUM> degrees clockwise from a printed symbol 106B. In this way, as a robot (e.g., robot <NUM>) traverses the agricultural area <NUM> and scans one or more fiducial markings <NUM>, the robot can determine: a direction in which the synthetic mulch <NUM> is extending, an orientation of multiple different strips of synthetic mulch <NUM>, and/or a location of the robot relative to the fiducial markings <NUM> and/or plants <NUM> (shown in <FIG>).

In some implementations, fiducial markings <NUM> can be disposed over one or more other features of an agricultural area <NUM> such as, irrigation drip tape, tubing, ground cover, nursery pots, silage bags, fumigation films, soil films, sensors, and/or any other apparatus or feature of an agricultural area. One or more fiducial markings <NUM> can be correlated with properties and/or information associated with the agricultural area <NUM>. Some properties and/or information can include, but are not limited to, any relevant information that can be correlated with one or more plants in the agricultural area. For instance, a fiducial marking can be correlated with an identifier for a nearby plant, and as the plant grows and yields produce, certain information such as plant type, plant age, yield, location, and/or any other data can be stored in association with the identifier for the plant and/or one or more nearby fiducial markings.

In some implementations, fiducial markings <NUM> can be disposed over features of the agricultural area <NUM> and the fiducial markings <NUM> can be visible from aerial vehicles such as drones, planes, rockets, satellites, and/or any other apparatus that can fly over the agricultural area. As an example, one or more rows of fiducial markings <NUM> can collectively represent one or more symbols that are visible from an aerial perspective (e.g., from one or more vision components operatively coupled to a satellite). In this way, one or more individual fiducial markings <NUM> can be used for identifying individual plants, whereas the one or more symbols created by the collection of fiducial markings <NUM> can be used for distinguishing a row and/or a field of the plants from another field of other plants.

In some cases, when a particular agricultural area is ready for seeds, the vehicle <NUM> and/or another agricultural device can create an opening <NUM> in portions of the synthetic mulch <NUM> in order to provide a space in which to plant one or more seeds, as illustrated in view <NUM> of <FIG>. Alternatively, seeds may be planted in the soil first, and then synthetic mulch <NUM> may be laid on top of the soil. This may enable the synthetic mulch to, for instance, condition the soil by heating it, allowing certain wavelengths of light through, etc. A location for an opening <NUM> can be determined based on properties of soil beneath the synthetic mulch <NUM>, as detected by one or more sensors embedded in the synthetic mulch <NUM> and/or in communication with the vehicle <NUM>. Alternatively, or additionally, openings <NUM> can be disposed about the synthetic mulch <NUM>, regardless of locations of fiducial markings <NUM>, at least based on the fiducial markings <NUM> having a variety of unique markings, thereby providing a surplus of markings that can be associated with each plant that will eventually protrude through the opening <NUM> as the plant grows.

<FIG> illustrates a view <NUM> of plants <NUM> growing through openings <NUM> of the synthetic mulch <NUM> that includes the fiducial markings <NUM>. In some implementations, properties of the synthetic mulch <NUM> can provide various benefits to the plants <NUM> as they grow and yield produce. For example, the synthetic mulch <NUM> can have optical properties capable of filtering certain frequencies of light such as ultraviolet, visible, near-infrared, middle infrared, and/or any other range or ranges of frequencies of the electromagnetic spectrum. Alternatively, or additionally, the synthetic mulch <NUM> can exhibit properties that may increase produce yield, increase produce size, shorten growth time, provide an anti-drip effect, provide an anti-fog effect, exhibit ultra-thermic properties, reduce water loss, prevent insect or pest infestation, mediate photosynthetically active radiation, maintaining humidity, maintain certain features of soil such as maintaining methyl bromide within the soil, and/or any other property that can improve health and/or quality of natural organisms within an agricultural area <NUM>.

Although the openings <NUM> provided in <FIG> are surrounded by and/or otherwise proximate to fiducial markings <NUM>-in some implementations, an opening <NUM> may not be entirely surrounded by fiducial markings <NUM>, but rather, can be adjacent to one or more fiducial markings <NUM>. As noted previously, each fiducial marking <NUM> can include a QR code, a barcode, natural language content, and/or any other marking that can be uniquely generated by a computing device. In some implementations, each fiducial marking <NUM> can be correlated to an identifier, such as data stored at a remote computing device or server, and the identifier can be correlated to an owner of the agricultural area <NUM> and/or a date in which the synthetic mulch <NUM> was provided at, and/or removed from, the agricultural area <NUM>. In this way, should synthetic mulch <NUM> be removed and disposed of, the synthetic mulch <NUM> can be later-identified in order to verify that the synthetic mulch <NUM> has been properly recycled and/or otherwise disposed of properly.

<FIG> and <FIG> illustrate views of a robot <NUM> traversing an agricultural area <NUM> in order to identify locations of fiducial markings <NUM> relative to plants <NUM>, and later, correlate features of the plants <NUM> to the fiducial marking <NUM>. <FIG> illustrates a view <NUM> of a robot <NUM> traversing an agricultural area <NUM> in furtherance of correlating one or more plants <NUM> to one or more fiducial markings <NUM>. The robot <NUM> can include one or more apparatuses for inspecting and/or sensing certain features and/or properties of the agricultural area <NUM>, the plants <NUM>, and/or the synthetic mulch <NUM>. As the plants <NUM> grow through openings <NUM> of the synthetic mulch <NUM>, the robot <NUM> can generate agricultural data that provides a correlation between one or more fiducial markings <NUM> and a particular plant <NUM>. For example, the vehicle <NUM> (shown in <FIG>) can print each fiducial marking <NUM> uniquely onto strips of synthetic mulch <NUM>, and, when the robot <NUM> scans a particular fiducial marking <NUM>, such as the particular fiducial marking <NUM>, the robot <NUM> can generate and/or identify a unique identifier that can be correlated to the scanned fiducial marking <NUM>.

As the robot <NUM> traverses the agricultural area <NUM> and scans more of the fiducial markings <NUM>, relative locations of the fiducial markings <NUM> can be identified and stored as data in correspondence with each generated identifier. For example, because the indicia printed at each fiducial marking <NUM> can be unique across an area of the fiducial marking <NUM>, a direction in which the robot <NUM> is scanning can be inferred from the fiducial markings <NUM>. Furthermore, in some implementations, the robot <NUM> can scan a strip of synthetic mulch <NUM> extending near a first side of the robot <NUM> as the robot <NUM> is traversing the agricultural area <NUM> and also scan fiducial markings <NUM> on another strip of synthetic mulch <NUM> extending near a second side of the robot <NUM> that is opposite the first side. In this way, each relative location of each fiducial marking <NUM> on the same area of synthetic mulch and/or different areas of synthetic mulch can be identified and stored as relative location data.

In some implementations, one or more image processing techniques and/or machine learning can be used in order to correlate each plant <NUM> to one or more fiducial markings <NUM>. For example, as illustrated in view <NUM> of <FIG>, the robot <NUM> can scan one or more portions of a plant <NUM> in order to generate data from which properties and/or features of the plant <NUM> can be determined. In some implementations, the robot <NUM> can scan one or more fiducial markings <NUM> simultaneous to scanning one or more plants <NUM>. Thereafter, during processing of one or more images generated from the scanning, the one or more fiducial markings <NUM> can be correlated to the one or more plants <NUM>. Furthermore, a location of each plant <NUM> can be identified based on a relative location of the fiducial markings <NUM> captured from one or more scans by the robot <NUM>.

In some implementations, locations of the plants <NUM> and/or the fiducial markings <NUM> can be determined using differential GPS. This can be useful in some implementations when, for example, the plants <NUM> are growing under a canopy that causes satellite GPS to be inaccurate and/or ineffective for determining locations of the plants <NUM> and/or the robot <NUM>. When using differential GPS, one or more reference stations can broadcast a signal characterizing a known location for the particular reference station. One more robots <NUM> traversing the agricultural area <NUM> can receive signals from one or more reference stations in order to calculate an exact location for a robot <NUM> at a particular time and/or location. When a current location is determined for a robot <NUM> that is scanning a particular fiducial marking <NUM> and/or plant <NUM>, the robot <NUM> and/or another computing device can generate location data that can be stored in association with the fiducial marking <NUM> and/or the plant <NUM>, in furtherance of tracking changes exhibited by the plant <NUM>. Alternatively, or additionally, the robot <NUM> can refrain from employing GPS and/or RTK tracking techniques. Rather, the robot <NUM> can track and/or manage data characterizing relative locations of fiducial markings <NUM> with respect to other fiducial markings <NUM>. For example, the robot <NUM> can generate data characterizing x, y, and/or z coordinates of fiducial markings <NUM> in order to determine their relative position and/or distance from each other. In this way, the robot <NUM> can generate routes to plants <NUM> of interest by inspecting nearby fiducial markings <NUM> in order to determine relative location of other fiducial markings <NUM> that are more proximate to the plants <NUM> of interest.

In other implementations, robot <NUM> may localize solely based on fiducial markings <NUM>. For example, after synthetic mulch is applied to agricultural area <NUM>, one or more robots may scout the agricultural area <NUM> surveying locations of fiducial markings <NUM> and, where applicable, the locations of fiducial markings <NUM> relative to plants <NUM>. These robots may store this initial scouted data in one or more databases. Subsequent robots traveling through agricultural area <NUM> may use this data to localize themselves relative to plants <NUM>. Consequently, the subsequent robots do not need to rely on GPS, which as noted previously may not be sufficiently accurate or even available (e.g., under a natural or man-made canopy) to determine which individual plant a robot is proximate.

The data stored in relation to each fiducial marking <NUM> and/or each plant <NUM> can be used prior to planting seeds, during planting or transplanting, during plant growth, during harvesting, collecting of produce from the plants <NUM>, and/or when produce is received by a third party and/or consumed by consumers. For example, for each plant <NUM>, agricultural data can be generated and stored in correlation with an identifier associated with the plant and/or a nearby fiducial marking(s) <NUM>. Such data can include yield estimation, phenotyping, disease characteristics, plant characteristics such as color, chemical makeup, soil quality, temperature, moisture, brightness, surface texture, yield, growth rate, and/or any other data that can be associated with a plant. When a plant has yielded some amount of produce and the produce is received by a third party entity, the produce can be inspected at a receiving facility in order to determine certain properties of the produce.

For example, images of the received produce can be captured and processed to determine whether any of the produce is diseased. The diseased produce can be discarded and information provided to the supplier can be used to identify an origin of the diseased produce. For instance, printed indicia on the produce and/or a container carrying the produce can be correlated to one or more fiducial markings <NUM> disposed over the synthetic mulch <NUM> through which the yielding plant grew. Because each fiducial marking <NUM> can be associated with location data, the robot <NUM> can determine that the third party entity detected diseased produce that came from at a particular location within the agricultural area <NUM>, navigate to the particular location that corresponds to the fiducial marking for the suspect plant, and perform one or more operations in furtherance of curing the plant of any disease.

For example, as illustrated in <FIG>, the robot <NUM> can perform an operation of providing some amount of material to an area from which a particular plant <NUM> is growing. As an example, when images of the received produce indicate that the yielding plant is under-watered, the robot <NUM> can process data received from the third party entity and determine that an operation of watering should be performed at the location corresponding to the under-watered plant <NUM>. In this way, the robot <NUM> can be proactive about curing deficiencies of produce yielding plants in response to third party entities (e.g., grocery stores) detecting issues with certain produce. This can allow the supplier of the produce to yield larger sums of produce, as many issues would be cured more quickly thereby reducing a probability that more produce will be effectively yielded.

<FIG> illustrates a system <NUM> for performing a variety of different agricultural operations using fiducial markings as reference points for tracking and/or initializing particular operations. Specifically, the system <NUM> can include a computing device <NUM>, which can be a robot capable of navigating through an agricultural area that can be used for yielding produce. For example, the computing device <NUM> can include one or more sensors <NUM> and/or various operational hardware <NUM> (e.g., end effectors, wheels, tracks, propellers, motors, etc.) that can be operated to enable the computing device <NUM> to autonomously navigate through an agricultural area, identify fiducial markings and/or plants, perform operations in furtherance of maintaining the agricultural area and/or the plants, collecting produce, and/or any other operation that can be performed at, or associated with, an agricultural area.

When the computing device <NUM> is navigating through an agricultural area that includes synthetic mulch with fiducial markings, the computing device <NUM> can use one or more sensors <NUM> to scan one or more of the fiducial markings. Signals from the one or more sensors <NUM> can be communicated to a fiducial processing engine <NUM>. The one or more sensors <NUM> can include a camera, barcode scanner, QR code scanner, RFID scanner, and/or any other sensor(s) that can be used to scan an identifier for an object. The fiducial processing engine <NUM> can process signals from the sensors <NUM> in order to identify one or more particular fiducial markings that the computing device <NUM> has scanned. For example, the fiducial processing engine <NUM> can generate or otherwise identify an identifier for each fiducial marking scanned by the computing device <NUM>. The fiducial processing engine <NUM> can use correlation data <NUM> in order to determine a correspondence between one or more fiducial markings and one or more identifiers stored in a database and/or plants in the agricultural area.

In some implementations, the fiducial processing engine <NUM> can use the correlation data <NUM> in order to determine a location and/or trajectory for the computing device <NUM>, as the computing device <NUM> traverses a portion of the agricultural area. For example, the fiducial processing engine <NUM> can determine an orientation of the fiducial markings (e.g., matrix barcodes have inherent orientations) relative to previously scanned fiducial markings in order to determine a current location and/or trajectory of the computing device <NUM>. The current location and/or the trajectory can be communicated to a route navigation engine <NUM>, which can assist with providing instructions for navigating the computing device <NUM> through a particular route. For instance, the route navigation engine <NUM> can communicate with a hardware controls engine <NUM> in order to control operational hardware <NUM> of the computing device <NUM> in furtherance of controlling certain navigational aspects of the computing device <NUM> such as, velocity, steering, acceleration, lift, yaw, roll, and/or any other navigational maneuver capable of being performed by the computing device <NUM>.

In some implementations, when the fiducial processing engine <NUM> has determined a correlation between a particular plant and one or more fiducial markings, the fiducial processing engine <NUM> can communicate with an agricultural data engine <NUM> in order to determine whether to perform one or more operations at the particular plant. For example, the agricultural data engine <NUM> can access agricultural data <NUM> and/or historical data <NUM> to determine whether the particular plant has a history of exhibiting one or more deficiencies. In some implementations, an environmental analysis engine <NUM> can be in communication with the one or more sensors <NUM> in order to determine whether a particular plant, which has been scanned by the one or more sensors <NUM>, is exhibiting a feature or property indicative of a deficiency for the particular plant. When the environmental analysis engine <NUM> has identified a particular deficiency, the environmental analysis engine <NUM> can communicate with the agricultural data engine <NUM>, which can generate agricultural data characterizing the particular deficiency. The generated agricultural data can be stored with the agricultural data <NUM>, and correlation data can be generated by a data correlation engine <NUM> in order to characterize the correlation between the particular deficiency, and the particular plant and/or fiducial marking that was scanned.

When a particular deficiency is identified, data characterizing the particular deficiency can be accessed by the hardware controls engine <NUM> in order for the hardware controls engine <NUM> to select a particular operation to treat or cure of the particular deficiency. For example, the operational hardware <NUM> can include one or more apparatuses for providing to, and/or removing from, a particular plant, certain materials based on the particular deficiency identified. For instance, when the hardware controls engine <NUM> determines that the particular deficiency corresponds to a lack of minerals or lack of water, the hardware controls engine <NUM> can cause the operational hardware <NUM> to provide minerals and/or water to the particular plant exhibiting the particular deficiency.

In some implementations, a third party entity <NUM> can be in communication with the computing device <NUM> in order to determine properties of the produce received from an agricultural area that the computing device <NUM> is operating to maintain. For example, the third party entity <NUM> can communicate received produce data <NUM> characterizing properties and/or features of produce picked up by the computing device <NUM> and/or another device that is associated with the computing device <NUM>. The environmental analysis engine <NUM> can process the received produce data <NUM> in order to identify a particular deficiency associated with some amount of produce supplied to the third party entity <NUM>. Furthermore, the data correlation engine <NUM> can use the received produce data <NUM> and historical data <NUM> in order to correlate the received produce data <NUM> to one or more plants that have historically yielded produce that was provided to the third party entity <NUM>.

When the environmental analysis engine <NUM> has identified one or more deficiencies or other ailments causing the defective produce, and when the data correlation engine <NUM> has identified the one of more plants that yielded the defective produce, the route navigation engine <NUM> can, based on this identification, generate a route for navigating to the one or more plants. Furthermore, the hardware controls engine <NUM> can identify one or more operations to perform at the one or more plants in order to improve a quality or condition of the one or more plants. The hardware controls engine <NUM> can then provide instructions to the operational hardware <NUM> in furtherance of causing the one or more operations to be performed on the one or more plants. In some implementations, feedback from the operational hardware <NUM> and/or the sensors <NUM> can be used to collect data, which can be indicative of an efficacy of operations performed on the one or more plants to improve their quality and/or condition. In this way, the computing device <NUM> will be able to learn and adapt to more frequently employ those operations that have a history of effectively improving plants that have been indicated as deficient by third party entities <NUM>.

<FIG> illustrates unclaimed method <NUM> for identifying fiducial markings located in an agricultural area and generating data for each fiducial marking and each plant that corresponds to a fiducial marking. The method <NUM> can be performed by one or more computing devices, applications, and/or any other apparatus or module capable of controlling a robot and/or accessing agricultural data. The method <NUM> can include an operation <NUM> of determining whether there has been a local or environmental change at a plant in an agricultural area. In some implementations, the determination at operation <NUM> can be based on information received from a separate entity that scans produce from the agricultural area for deficiencies, such as certain physical properties that can make the produce unhealthy for consumers. Alternatively, or additionally, the determination at operation <NUM> can be based on information received from a robot that has traversed a portion of the agricultural area that includes the plant and has detected a local change at the plant using one or more sensors.

When a local or environmental change is not determined to have been detected at the plant in the agricultural area, the method <NUM> can continue to monitor for local and/or environmental changes. However, when a local or environmental change is determined to have been detected at the plant in the agricultural area, the method <NUM> can proceed from the operation <NUM> to an operation <NUM>. The operation <NUM> can include identifying a fiducial marking corresponding to the affected plant. In some implementations, identifying the fiducial marking can include processing the received information to determine a correlation between the received information and the plants or the fiducial marking corresponding to the plant. Alternatively, or additionally, identifying the fiducial marking can include processing data generated by a robot using one or more sensors that detect the fiducial markings at or near a plant. The fiducial markings can be, for example, a barcode, a QR code, an RFID tag, a printed label, an embedded label, and/or any other identifier that can be located on a synthetic mulch or other agricultural surface.

In some implementations, data stored in correlation with a fiducial marking can include coordinate data that characterizes a latitude and/or longitude of a position of the fiducial marking, a time stamp that characterizes a time and/or date on which the fiducial marking was scanned, and/or any other metadata that can characterize properties of a circumstance in which the fiducial marking was scanned. Thereafter, other fiducial markings can be determined to be related based on similarities and/or differences between metadata. For instance, when multiple fiducial markings are determined to be in close proximity to one another when a plant and any surrounding fiducial markings are inspected, unique identifiers corresponding to those fiducial markings can be correlated, in a database, to the previously generated metadata. In this way, agricultural areas of interest to the robot and other entities can be identified via correspondence between metadata correlated to the physical fiducial markings disposed about an agricultural area.

The method <NUM> can proceed from the operation <NUM> to an operation <NUM>. The operation <NUM> can include generating data that characterizes the local or environmental change, and/or an effect on the plant. As an example, as the robot is navigating through the agricultural area and has identified the particular plant, the robot can use one or more sensors to identify one or more physical properties of the particular plant. Thereafter, the fiducial marking can be scanned and the robot can generate data that provides a correspondence between the physical properties and the fiducial marking. Alternatively, or additionally, the information provided by a separate entity that has received produce from the plant can indicate a local or environmental change to the particular plant based on scanning the produce. Using this information, the robot can navigate to the fiducial marking corresponding to the plant, and use one or more sensors to identify physical properties of the particular plant. The robot can then, at operation <NUM>, optionally generate data characterizing the physical properties and store the generated data in association with the fiducial marking and/or the particular plant. Specifically, the method <NUM> can proceed from the operation <NUM> to the operation <NUM> of optionally storing the generated data in association with the fiducial marking and/or the affected plant.

The method <NUM> can proceed from the operation <NUM> to an operation <NUM> that includes determining whether a route being navigated by the robot has been completed. The route can be stored at the robot and can be used by the robot to move from fiduciary marking to fiduciary marking. For example, route data can characterize a location of each plant in the agricultural area relative to each fiduciary marking in the agricultural area. In this way, when the robot identifies a particular fiduciary marking and a plant, the robot will be able to identify a relative location of any other fiduciary marking and/or other plants.

When the route is determined to not have been completed by the robot, the method <NUM> can proceed from the operation <NUM> to an operation <NUM> of continuing navigating through the route of the agricultural area. Alternatively, when the route is determined to have been completed by the robot, the method <NUM> can proceed from the operation <NUM> to an operation <NUM>. The operation <NUM> can include determining whether to initialize another navigation routine, in which the robot will maneuver through the same route or a different route through the agricultural area. For example, when the robot completes a route of scanning each fiduciary marking and optionally generating data for each plant, the robot can travel out of the agricultural area and dock itself in order to receive maintenance, receive charge, upload data, receive updates, and/or perform any other operation that may be associated with the robot and/or the agricultural area. Thereafter, and at the operation <NUM>, when the robot is instructed to, or otherwise caused to, initialize another navigation routine, the method <NUM> can proceed from the operation <NUM> to the operation <NUM>. Furthermore, as the robot is navigating through the other routine, the robot can use one or more sensors to detect local and environmental changes in accordance with the operation <NUM>.

<FIG> illustrates a method <NUM> for correlating produce data to individual plants from which corresponding produce came. The method <NUM> can be performed by one or more computing devices, applications, and/or any other apparatus or module capable of controlling a robot and/or accessing agricultural data. The method <NUM> can include an operation <NUM> of navigating through a route within an agricultural area. The route can be characterized by route data stored at a robot that operates to monitor changes to plants located within the agricultural area. When the robot detects a plant property, physical feature, and/or particular condition of a plant that is helpful or problematic to a total yield of the plant, the robot can generate agricultural data characterizing the property, feature, and/or the condition. Furthermore, the robot can address certain properties and/or conditions that can be remedied by one or more operations of the robot, such as adding or removing materials or organisms to or from an area of one or more plants of a plurality of plants located within the agricultural area.

The method <NUM> can proceed from the operation <NUM> to an operation <NUM> of identifying a fiducial marking located near a plant in the agricultural area. The fiducial marking can be a label that is disposed over synthetic mulch that at least partially surrounds the plant in the agricultural area. The label can include any amount of indicia, such as one or more symbols, numbers, letters, barcodes, texture, and/or any combination thereof. Alternatively, or additionally, the fiducial marking can be an RFID tag that is attached to the synthetic mulch and/or a location adjacent to each plant of the plurality of plants the robot can identify a fiducial marking by using one or more sensors to collect data from the fiducial marking and process the data to confirm that the collected data corresponds to a particular fiducial marking.

The method <NUM> can proceed from the operation <NUM> to an operation <NUM> that includes identifying correspondence data that provides a correlation between produce and a fiducial marking. The correspondence data can be generated based on previous instances when one or more robots traversed the agricultural area, identified various fiducial markings and nearby plants, and generated the correspondence data in order to characterize relationships between the fiducial markings and the plants. Alternatively, or additionally, the correspondence data can be provided by one or more computing devices tasked with scanning an area that includes the plants and the fiducial markings, processing images that include the plants and of the fiducial markings, and generating the correspondence data that is based on the processing.

The method <NUM> can proceed from the operation <NUM> to an operation <NUM> that includes determining whether a recipient of produce detected a deficiency with the produce that was supplied from the agricultural area. The determination can be based on information supplied by the recipient, which can use one or more devices to examine produce supplied from the agricultural area. For example, one or more computing devices can be used by the recipient to process images of incoming produce, and the images can be analyzed using one or more machine learning models that have been trained to identify problematic or otherwise deficient vegetation. Based on the analysis, the recipient of the produce can transmit information to the robot and/or any other computing device associated with the agricultural area. In this way, the robot and/or other computing devices can remedy one more plants that are exhibiting the deficiency identified by the recipient of the produce.

When a recipient of the produce has not identified a deficiency with the produce supplied by the plant corresponding to the identified fiducial marking, the method <NUM> can proceed from the operation <NUM> to an operation <NUM>. The operation <NUM> can be an optional operation that includes generating data characterizing one or more properties of the plant. For example, despite the plant corresponding to the identified fiducial marking not having a record of providing sufficient produce, the robot can none the less analyze the plant and generate data characterizing one or more properties of the plant. In this way, should one or more properties characterize an unhealthy property of the plant, the operations taken by the robot would prevent subsequent harm, should produce from the plant be disseminated.

When a recipient of the produce has identified a deficiency with the produce supplied by the plant corresponding to the identified fiducial marking, the method <NUM> can proceed from the operation <NUM> to an operation <NUM>. The operation <NUM> can include generating data characterizing a property of the plant that provided the deficient produce. In this way, the robot can verify whether there is a detectable deficiency with the plant. Moreover, the robot can generate the data in order to provide some correspondence between a current state at the plant and the deficiency of the produce provided to the recipient. In some implementations, data generated by the robot and information provided by a recipient can be used to train one or more machine learning models in order to allow each robot to use subsequently collected data to diagnose and/or remedy certain conditions of plants.

In some implementations, the method <NUM> can optionally include an operation <NUM> of performing one or more operations to remedy the condition of the plant that provided the deficient produce. For example, when a condition of the plant that caused the deficient produce corresponds to an organism that negatively impacted an environment of the produce, the robot can perform one or more operations to remove and/or eliminate the organism. Alternatively, or additionally, when a condition of the plant that yielded the produce corresponds to a lack of minerals and/or water, the robot can perform one or more operations to add water and/or other materials to an area occupied by the plant. For instance, the robot can maneuver toward the plant and add or remove materials or organisms to, or from, an area defined by an outer edge of the plant. Furthermore, the robot can generate data characterizing the one or more operations performed at the particular plant for future reference and/or for determining whether the one or more operations were effective in treating the particular plant. Thereafter, the data can be used during subsequent operations to adjust properties of materials added to and/or removed from an area occupied by a plant, based on historically recorded efficacy of the properties treating plants during previous operations. The method <NUM> can proceed to an operation <NUM> of storing any generated data in association with the fiducial marking and the particular plant.

The method <NUM> can proceed from the operation <NUM> to an operation <NUM> that includes determining whether a route through which the robot is navigating has been completed. When the robot has not completed the current route, the method <NUM> can proceed from the operation <NUM> to the operation <NUM>, in furtherance of traversing the agricultural area toward a different plant and/or a different fiducial marking. However, when the route has been completed by the robot, the method <NUM> can proceed from the operation <NUM> to an operation <NUM>. The operation <NUM> can include determining whether to initialize another navigation routine. Instructions to perform another navigation routine can be autonomously identified and/or received from a source that is external to the robot. When the robot has determined to initialize another navigation routine, the method <NUM> can proceed to the operation <NUM>, otherwise the robot can await further instructions to assist with certain operations in the agricultural area.

<FIG> schematically depicts an example architecture of a robot <NUM>. The robot <NUM> includes a robot control system <NUM>, one or more operational components 640A-640N, and one or more sensors 642A-<NUM>. The sensors 642A-<NUM> may include, for example, vision components, light sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors, accelerometers, gyroscopes, thermometers, barometers, and so forth. While sensors 642A-<NUM> are depicted as being integral with robot <NUM>, this is not meant to be limiting. In some implementations, sensors 642A-<NUM> may be located external to robot <NUM>, e.g., as standalone units.

Operational components 640A-640N may include, for example, one or more end effectors and/or one or more servo motors or other actuators to effectuate movement of one or more components of the robot. For example, the robot <NUM> may have multiple degrees of freedom and each of the actuators may control actuation of the robot <NUM> within one or more of the degrees of freedom responsive to the control commands. As used herein, the term actuator encompasses a mechanical or electrical device that creates motion (e.g., a motor), in addition to any driver(s) that may be associated with the actuator and that translate received control commands into one or more signals for driving the actuator. Accordingly, providing a control command to an actuator may comprise providing the control command to a driver that translates the control command into appropriate signals for driving an electrical or mechanical device to create desired motion.

The robot control system <NUM> may be implemented in one or more processors, such as a CPU, GPU, and/or other controller(s) of the robot <NUM>. In some implementations, the robot <NUM> may comprise a "brain box" that may include all or aspects of the control system <NUM>. For example, the brain box may provide real time bursts of data to the operational components 640A-640N, with each of the real time bursts comprising a set of one or more control commands that dictate, inter alia, the parameters of motion (if any) for each of one or more of the operational components 640A-640N. In some implementations, the robot control system <NUM> may perform one or more aspects of one or more methods described herein.

As described herein, in some implementations all or aspects of the control commands generated by control system <NUM> can be generated based on 3D bounding shapes generated according to techniques described herein. Although control system <NUM> is illustrated in <FIG> as an integral part of the robot <NUM>, in some implementations, all or aspects of the control system <NUM> may be implemented in a component that is separate from, but in communication with, robot <NUM>. For example, all or aspects of control system <NUM> may be implemented on one or more computing devices that are in wired and/or wireless communication with the robot <NUM>, such as computer system <NUM>.

Storage subsystem <NUM> stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem <NUM> may include the logic to perform selected aspects of method <NUM>, method <NUM> and/or to implement one or more of system <NUM>, vehicle <NUM>, embedded apparatuses in synthetic mulch, robot <NUM>, and/or any other application, device, apparatus, and/or module discussed herein.

Claim 1:
A method implemented by one or more processors, the method comprising:
causing one or more robots (<NUM>) to identify, in an agricultural area (<NUM>; <NUM>) that includes a plurality of plants (<NUM>; <NUM>) each extending through an aperture (<NUM>) in a synthetic mulch (<NUM>; <NUM>) that is disposed over the agricultural area and that includes fiducial markings (<NUM>; <NUM>) that are detectable by a robot, each fiducial marking in the agricultural area and to generate correspondence data that correlates each plant of the plurality of plants to one or more fiducial markings; and
subsequent to causing the one or more robots to identify each fiducial marking:
causing one or more of the robots to navigate through the agricultural area in furtherance of locating a particular plant of the plurality of plants using the correspondence data, and
when the one or more robots have located the particular plant:
causing the one or more robots to perform an operation in furtherance of affecting a physical condition of the plant.