Patent Description:
Farming machines generally outsize human operators and various other objects in their environment. With such bulk in size, such farming machines generally have difficulty gauging the proximity of objects to the farming machines. As such, operators must move slowly and with great care, especially in confined or dense environments. The challenge in navigating such environments is further compounded in the case with automated farming machines.

According to a first aspect of the disclosure there is provided a method for establishing a virtual safety bubble surrounding an autonomous farming machine, a non-transitory computer-readable storage medium storing instructions for establishing a virtual safety bubble surrounding an autonomous farming machine, and an autonomous farming machine, as set out in the attached claims.

Various autonomous vehicles and related sensor systems are also discussed in each of <CIT>, <CIT>, and in<NPL>.

A farming machine is configured to generate and maintain a virtual safety bubble that enables the farming machine to autonomously perform farming actions in the environment safely. The farming machine is, generally, a navigable vehicle comprising a detection system, a treatment mechanism, a verification mechanism, among other components. The detection system includes one or more detection mechanisms, e.g., cameras, for capturing image data of the surrounding environment. The detection system may have a <NUM>-degree field of view, aggregated from the individual fields of view of the detection mechanisms. The farming machine's treatment mechanism applies treatment as part of the performable farming action(s) and the verification mechanism verifies that the treatment was successful. The farming machine may further comprise a control system, e.g., a general computing system, for controlling operation of the various components.

To aid in the safe navigation of the farming machine, the control system may generate and maintain a virtual safety bubble that triggers when an obstacle breaches the virtual safety bubble. When an object is determined to have breached, i.e., is within the virtual safety bubble, the control system may terminate or cease operations, and/or may enact other preventive measures. Preventive measures include rerouting the farming machine around the obstacle, changing a configuration of the farming machine, requesting input from an operator or a manager, etc..

The control system generates the virtual safety bubble based on a configuration of the farming machine. As the farming machine changes configuration, the control system can automatically and/or dynamically adjust the virtual safety bubble, by adjusting characteristics of the virtual safety bubble. For example, when the farming machine accelerates to a higher velocity, the control system can automatically and/or dynamically adjust a size of the virtual safety bubble to be larger than before to provide additional distance to enact the preventive measures. In another example, the farming machine may change its configuration to perform different farming actions, and the control system can automatically and/or dynamically adjust the parameters of the virtual safety bubble in response to the changed configuration.

Agricultural managers ("managers") are responsible for managing farming operations in one or more fields. Managers work to implement a farming objective within those fields and select from among a variety of farming actions to implement that farming objective. Traditionally, managers are, for example, a farmer or agronomist that works the field but could also be other people and/or systems configured to manage farming operations within the field. For example, a manager could be an automated farming machine, a machine learned computer model, etc. In some cases, a manager may be a combination of the managers described above. For example, a manager may include a farmer assisted by a machine learned agronomy model and one or more automated farming machines or could be a farmer and an agronomist working in tandem.

Managers implement one or more farming objectives for a field. A farming objective is typically a macro-level goal for a field. For example, macro-level farming objectives may include treating crops with growth promotors, neutralizing weeds with growth regulators, harvesting a crop with the best possible crop yield, or any other suitable farming objective. However, farming objectives may also be a micro-level goal for the field. For example, micro-level farming objectives may include treating a particular plant in the field, repairing or correcting a part of a farming machine, requesting feedback from a manager, etc. Of course, there are many possible farming objectives and combinations of farming objectives, and the previously described examples are not intended to be limiting.

Farming objectives are accomplished by one or more farming machines performing a series of farming actions. Farming machines are described in greater detail below. Farming actions are any operation implementable by a farming machine within the field that works towards a farming objective. Consider, for example, a farming objective of harvesting a crop with the best possible yield. This farming objective requires a litany of farming actions, e.g., planting the field, fertilizing the plants <NUM>, watering the plants <NUM>, weeding the field, harvesting the plants <NUM>, evaluating yield, etc. Similarly, each farming action pertaining to harvesting the crop may be a farming objective in and of itself. For instance, planting the field can require its own set of farming actions, e.g., preparing the soil, digging in the soil, planting a seed, etc..

In other words, managers implement a treatment plan in the field to accomplish a farming objective. A treatment plan is a hierarchical set of macro-level and/or micro-level objectives that accomplish the farming objective of the manager. Within a treatment plan, each macro or micro-objective may require a set of farming actions to accomplish, or each macro or micro-objective may be a farming action itself. So, to expand, the treatment plan is a temporally sequenced set of farming actions to apply to the field that the manager expects will accomplish the farming objective.

When executing a treatment plan in a field, the treatment plan itself and/or its constituent farming objectives and farming actions have various results. A result is a representation as to whether, or how well, a farming machine accomplished the treatment plan, farming objective, and/or farming action. A result may be a qualitative measure such as "accomplished" or "not accomplished," or may be a quantitative measure such as "<NUM> pounds harvested," or "<NUM> acres treated. " Results can also be positive or negative, depending on the configuration of the farming machine or the implementation of the treatment plan. Moreover, results can be measured by sensors of the farming machine, input by managers, or accessed from a datastore or a network.

Traditionally, managers have leveraged their experience, expertise, and technical knowledge when implementing farming actions in a treatment plan. In a first example, a manager may spot check weed pressure in several areas of the field to determine when a field is ready for weeding. In a second example, a manager may refer to previous implementations of a treatment plan to determine the best time to begin planting a field. Finally, in a third example, a manager may rely on established best practices in determining a specific set of farming actions to perform in a treatment plan to accomplish a farming objective.

Leveraging manager and historical knowledge to make decisions for a treatment plan affects both spatial and temporal characteristics of a treatment plan. For instance, farming actions in a treatment plan have historically been applied to entire field rather than small portions of a field. To illustrate, when a manager decides to plant a crop, she plants the entire field instead of just a corner of the field having the best planting conditions; or, when the manager decides to weed a field, she weeds the entire field rather than just a few rows. Similarly, each farming action in the sequence of farming actions of a treatment plan are historically performed at approximately the same time. For example, when a manager decides to fertilize a field, she fertilizes the field at approximately the same time; or, when the manager decides to harvest the field, she does so at approximately the same time.

Notably though, farming machines have greatly advanced in their capabilities. For example, farming machines continue to become more autonomous, include an increasing number of sensors and measurement devices, employ higher amounts of processing power and connectivity, and implement various machine vision algorithms to enable managers to successfully implement a treatment plan.

Because of this increase in capability, managers are no longer limited to spatially and temporally monolithic implementations of farming actions in a treatment plan. Instead, managers may leverage advanced capabilities of farming machines to implement treatment plans that are highly localized and determined by real-time measurements in the field. In other words, rather than a manager applying a "best guess" treatment plan to an entire field, they can implement individualized and informed treatment plans for each plant in the field.

A farming machine that implements farming actions of a treatment plan may have a variety of configurations, some of which are described in greater detail below.

<FIG> is an isometric view of a farming machine <NUM> that performs farming actions of a treatment plan, according to one example embodiment, and <FIG> is a top view of the farming machine <NUM> in <FIG>. <FIG> is an isometric view of another farming machine <NUM> that performs farming actions of a treatment plan, in accordance with one or more embodiments.

The farming machine <NUM> generally includes a detection mechanism <NUM>, a treatment mechanism <NUM>, and a control system <NUM>. The farming machine <NUM> can additionally include a mounting mechanism <NUM>, a verification mechanism <NUM>, a power source, digital memory, communication apparatus, or any other suitable component that enables the farming machine <NUM> to implement farming actions in a treatment plan. Moreover, the described components and functions of the farming machine <NUM> are just examples, and a farming machine <NUM> can have different or additional components and functions other than those described below.

The farming machine <NUM> is configured to perform farming actions in a field <NUM>, and the implemented farming actions are part of a treatment plan. To illustrate, the farming machine <NUM> implements a farming action which applies a treatment to one or more plants <NUM> and/or the substrate <NUM> within a geographic area. Here, the treatment farming actions are included in a treatment plan to regulate plant growth. As such, treatments are typically applied directly to a single plant <NUM>, but can alternatively be directly applied to multiple plants <NUM>, indirectly applied to one or more plants <NUM>, applied to the environment <NUM> associated with the plant <NUM> (e.g., soil, atmosphere, or other suitable portion of the plant's environment adjacent to or connected by an environmental factors, such as wind), or otherwise applied to the plants <NUM>.

In a particular example, the farming machine <NUM> is configured to implement a farming action which applies a treatment that necroses the entire plant <NUM> (e.g., weeding) or part of the plant <NUM> (e.g., pruning). In this case, the farming action can include dislodging the plant <NUM> from the supporting substrate <NUM>, incinerating a portion of the plant <NUM> (e.g., with directed electromagnetic energy such as a laser), applying a treatment concentration of working fluid (e.g., fertilizer, hormone, water, etc.) to the plant <NUM>, or treating the plant <NUM> in any other suitable manner.

In another example, the farming machine <NUM> is configured to implement a farming action which applies a treatment to regulate plant growth. Regulating plant growth can include promoting plant growth, promoting growth of a plant portion, hindering (e.g., retarding) plant <NUM> or plant portion growth, or otherwise controlling plant growth. Examples of regulating plant growth includes applying growth hormone to the plant <NUM>, applying fertilizer to the plant <NUM> or substrate <NUM>, applying a disease treatment or insect treatment to the plant <NUM>, electrically stimulating the plant <NUM>, watering the plant <NUM>, pruning the plant <NUM>, or otherwise treating the plant <NUM>. Plant growth can additionally be regulated by pruning, necrosing, or otherwise treating the plants <NUM> adjacent to the plant <NUM>.

The farming machine <NUM> operates in an operating environment <NUM>. The operating environment <NUM> is the environment <NUM> surrounding the farming machine <NUM> while it implements farming actions of a treatment plan. The operating environment <NUM> may also include the farming machine <NUM> and its corresponding components itself.

The operating environment <NUM> typically includes a field <NUM>, and the farming machine <NUM> generally implements farming actions of the treatment plan in the field <NUM>. A field <NUM> is a geographic area where the farming machine <NUM> implements a treatment plan. The field <NUM> may be an outdoor plant field but could also be an indoor location that houses plants such as, e.g., a greenhouse, a laboratory, a grow house, a set of containers, or any other suitable environment <NUM>.

A field <NUM> may include any number of field portions. A field portion is a subunit of a field <NUM>. For example, a field portion may be a portion of the field <NUM> small enough to include a single plant <NUM>, large enough to include many plants <NUM>, or some other size. The farming machine <NUM> can execute different farming actions for different field portions. For example, the farming machine <NUM> may apply an herbicide for some field portions in the field <NUM>, while applying a pesticide in another field portion. Moreover, a field <NUM> and a field portion are largely interchangeable in the context of the methods and systems described herein. That is, treatment plans and their corresponding farming actions may be applied to an entire field <NUM> or a field portion depending on the circumstances at play.

The operating environment <NUM> may also include plants <NUM>. As such, farming actions the farming machine <NUM> implements as part of a treatment plan may be applied to plants <NUM> in the field <NUM>. The plants <NUM> can be crops but could also be weeds or any other suitable plant <NUM>. Some example crops include cotton, lettuce, soybeans, rice, carrots, tomatoes, corn, broccoli, cabbage, potatoes, wheat, or any other suitable commercial crop. The weeds may be grasses, broadleaf weeds, thistles, or any other suitable determinantal weed.

More generally, plants <NUM> may include a stem that is arranged superior to (e.g., above) the substrate <NUM> and a root system joined to the stem that is located inferior to the plane of the substrate <NUM> (e.g., below ground). The stem may support any branches, leaves, and/or fruits. The plant <NUM> can have a single stem, leaf, or fruit, multiple stems, leaves, or fruits, or any number of stems, leaves or fruits. The root system may be a tap root system or fibrous root system, and the root system may support the plant <NUM> position and absorb nutrients and water from the substrate <NUM>. In various examples, the plant <NUM> may be a vascular plant <NUM>, non-vascular plant <NUM>, ligneous plant <NUM>, herbaceous plant <NUM>, or be any suitable type of plant <NUM>.

Plants <NUM> in a field <NUM> may be grown in one or more plant <NUM> rows (e.g., plant <NUM> beds). The plant <NUM> rows are typically parallel to one another but do not have to be. Each plant <NUM> row is generally spaced between <NUM> inches and <NUM> inches apart when measured in a perpendicular direction from an axis representing the plant <NUM> row. Plant <NUM> rows can have wider or narrower spacings or could have variable spacing between multiple rows (e.g., a spacing of <NUM> in. between a first and a second row, a spacing of <NUM> in. a second and a third row, etc.).

Plants <NUM> within a field <NUM> may include the same type of crop (e.g., same genus, same species, etc.). For example, each field portion in a field <NUM> may include corn crops. However, the plants <NUM> within each field <NUM> may also include multiple crops (e.g., a first, a second crop, etc.). For example, some field portions may include lettuce crops while other field portions include pig weeds, or, in another example, some field portions may include beans while other field portions include corn. Additionally, a single field portion may include different types of crop. For example, a single field portion may include a soybean plant <NUM> and a grass weed.

The operating environment <NUM> may also include a substrate <NUM>. As such, farming actions the farming machine <NUM> implements as part of a treatment plan may be applied to the substrate <NUM>. The substrate <NUM> may be soil but can alternatively be a sponge or any other suitable substrate <NUM>. The substrate <NUM> may include plants <NUM> or may not include plants <NUM> depending on its location in the field <NUM>. For example, a portion of the substrate <NUM> may include a row of crops, while another portion of the substrate <NUM> between crop rows includes no plants <NUM>.

The farming machine <NUM> may include a detection mechanism <NUM>. The detection mechanism <NUM> identifies objects in the operating environment <NUM> of the farming machine <NUM>. To do so, the detection mechanism <NUM> obtains information describing the environment <NUM> (e.g., sensor or image data), and processes that information to identify pertinent objects (e.g., plants <NUM>, substrate <NUM>, persons, etc.) in the operating environment <NUM>. Identifying objects in the environment <NUM> further enables the farming machine <NUM> to implement farming actions in the field <NUM>. For example, the detection mechanism <NUM> may capture an image of the field <NUM> and process the image with a plant <NUM> identification model to identify plants <NUM> in the captured image. The farming machine <NUM> then implements farming actions in the field <NUM> based on the plants <NUM> identified in the image.

The farming machine <NUM> can include any number or type of detection mechanism <NUM> that may aid in determining and implementing farming actions. In some embodiments, the detection mechanism <NUM> includes one or more sensors. For example, the detection mechanism <NUM> can include a multispectral camera, a stereo camera, a CCD camera, a single lens camera, a CMOS camera, hyperspectral imaging system, LIDAR system (light detection and ranging system), a depth sensing system, dynamometer, IR camera, thermal camera, humidity sensor, light sensor, temperature sensor, or any other suitable sensor. Further, the detection mechanism <NUM> may include an array of sensors (e.g., an array of cameras) configured to capture information about the environment <NUM> surrounding the farming machine <NUM>. For example, the detection mechanism <NUM> may include an array of cameras configured to capture an array of pictures representing the environment <NUM> surrounding the farming machine <NUM>. The detection mechanism <NUM> may also be a sensor that measures a state of the farming machine <NUM>. For example, the detection mechanism <NUM> may be a speed sensor, a heat sensor, or some other sensor that can monitor the state of a component of the farming machine <NUM>. Additionally, the detection mechanism <NUM> may also be a sensor that measures components during implementation of a farming action. For example, the detection mechanism <NUM> may be a flow rate monitor, a grain harvesting sensor, a mechanical stress sensor etc. Whatever the case, the detection mechanism <NUM> senses information about the operating environment <NUM> (including the farming machine <NUM>).

A detection mechanism <NUM> may be mounted at any point on the mounting mechanism <NUM>. Depending on where the detection mechanism <NUM> is mounted relative to the treatment mechanism <NUM>, one or the other may pass over a geographic area in the field <NUM> before the other. For example, the detection mechanism <NUM> may be positioned on the mounting mechanism <NUM> such that it traverses over a geographic location before the treatment mechanism <NUM> as the farming machine <NUM> moves through the field <NUM>. In another examples, the detection mechanism <NUM> is positioned to the mounting mechanism <NUM> such that the two traverse over a geographic location at substantially the same time as the farming machine <NUM> moves through the filed. Similarly, the detection mechanism <NUM> may be positioned on the mounting mechanism <NUM> such that the treatment mechanism <NUM> traverses over a geographic location before the detection mechanism <NUM> as the farming machine <NUM> moves through the field <NUM>. The detection mechanism <NUM> may be statically mounted to the mounting mechanism <NUM>, or may be removably or dynamically coupled to the mounting mechanism <NUM>. In other examples, the detection mechanism <NUM> may be mounted to some other surface of the farming machine <NUM> or may be incorporated into another component of the farming machine <NUM>. The detection mechanism <NUM> may be removably coupled to the farming machine <NUM>.

The farming machine <NUM> may include a verification mechanism <NUM>. Generally, the verification mechanism <NUM> records a measurement of the operating environment <NUM> and the farming machine <NUM> may use the recorded measurement to verify or determine the extent of an implemented farming action (i.e., a result of the farming action).

To illustrate, consider an example where a farming machine <NUM> implements a farming action based on a measurement of the operating environment <NUM> by the detection mechanism <NUM>. The verification mechanism <NUM> records a measurement of the same geographic area measured by the detection mechanism <NUM> and where farming machine <NUM> implemented the determined farming action. The farming machine <NUM> then processes the recorded measurement to determine the result of the farming action. For example, the verification mechanism <NUM> may record an image of the geographic region surrounding a plant <NUM> identified by the detection mechanism <NUM> and treated by a treatment mechanism <NUM>. The farming machine <NUM> may apply a treatment detection algorithm to the recorded image to determine the result of the treatment applied to the plant <NUM>.

Information recorded by the verification mechanism <NUM> can also be used to empirically determine operation parameters of the farming machine <NUM> that will obtain the desired effects of implemented farming actions (e.g., to calibrate the farming machine <NUM>, to modify treatment plans, etc.). For instance, the farming machine <NUM> may apply a calibration detection algorithm to a measurement recorded by the farming machine <NUM>. In this case, the farming machine <NUM> determines whether the actual effects of an implemented farming action are the same as its intended effects. If the effects of the implemented farming action are different than its intended effects, the farming machine <NUM> may perform a calibration process. The calibration process changes operation parameters of the farming machine <NUM> such that effects of future implemented farming actions are the same as their intended effects. To illustrate, consider the previous example where the farming machine <NUM> recorded an image of a treated plant <NUM>. There, the farming machine <NUM> may apply a calibration algorithm to the recorded image to determine whether the treatment is appropriately calibrated (e.g., at its intended location in the operating environment <NUM>). If the farming machine <NUM> determines that the farming machine <NUM> is not calibrated (e.g., the applied treatment is at an incorrect location), the farming machine <NUM> may calibrate itself such that future treatments are in the correct location. Other example calibrations are also possible.

The verification mechanism <NUM> can have various configurations. For example, the verification mechanism <NUM> can be substantially similar (e.g., be the same type of mechanism as) the detection mechanism <NUM> or can be different from the detection mechanism <NUM>. In some cases, the detection mechanism <NUM> and the verification mechanism <NUM> may be one in the same.

(e.g., the same sensor). In an example configuration, the verification mechanism <NUM> is positioned distal the detection mechanism <NUM> relative the direction of travel <NUM>, and the treatment mechanism <NUM> is positioned there between. In this configuration, the verification mechanism <NUM> traverses over a geographic location in the operating environment <NUM> after the treatment mechanism <NUM> and the detection mechanism <NUM>. However, the mounting mechanism <NUM> can retain the relative positions of the system components in any other suitable configuration. In some configurations, the verification mechanism <NUM> can be included in other components of the farming machine <NUM>.

The farming machine <NUM> can include any number or type of verification mechanism <NUM>. In some embodiments, the verification mechanism <NUM> includes one or more sensors. For example, the verification mechanism <NUM> can include a multispectral camera, a stereo camera, a CCD camera, a single lens camera, a CMOS camera, hyperspectral imaging system, LIDAR system (light detection and ranging system), a depth sensing system, dynamometer, IR camera, thermal camera, humidity sensor, light sensor, temperature sensor, or any other suitable sensor. Further, the verification mechanism <NUM> may include an array of sensors (e.g., an array of cameras) configured to capture information about the environment <NUM> surrounding the farming machine <NUM>. For example, the verification mechanism <NUM> may include an array of cameras configured to capture an array of pictures representing the operating environment <NUM>.

The farming machine <NUM> may include a treatment mechanism <NUM>. The treatment mechanism <NUM> can implement farming actions in the operating environment <NUM> of a farming machine <NUM>. For instance, a farming machine <NUM> may include a treatment mechanism <NUM> that applies a treatment to a plant <NUM>, a substrate <NUM>, or some other object in the operating environment <NUM>. More generally, the farming machine <NUM> employs the treatment mechanism <NUM> to apply a treatment to a treatment area <NUM>, and the treatment area <NUM> may include anything within the operating environment <NUM> (e.g., a plant <NUM> or the substrate <NUM>). In other words, the treatment area <NUM> may be any portion of the operating environment <NUM>.

When the treatment is a plant treatment, the treatment mechanism <NUM> applies a treatment to a plant <NUM> in the field <NUM>. The treatment mechanism <NUM> may apply treatments to identified plants or non-identified plants. For example, the farming machine <NUM> may identify and treat a specific plant (e.g., plant <NUM>) in the field <NUM>. Alternatively, or additionally, the farming machine <NUM> may identify some other trigger that indicates a plant treatment and the treatment mechanism <NUM> may apply a plant treatment. Some example plant treatment mechanisms <NUM> include: one or more spray nozzles, one or more electromagnetic energy sources (e.g., a laser), one or more physical implements configured to manipulate plants, but other plant <NUM> treatment mechanisms <NUM> are also possible.

Additionally, when the treatment is a plant treatment, the effect of treating a plant <NUM> with a treatment mechanism <NUM> may include any of plant necrosis, plant growth stimulation, plant portion necrosis or removal, plant portion growth stimulation, or any other suitable treatment effect. Moreover, the treatment mechanism <NUM> can apply a treatment that dislodges a plant <NUM> from the substrate <NUM>, severs a plant <NUM> or portion of a plant <NUM> (e.g., cutting), incinerates a plant <NUM> or portion of a plant <NUM>, electrically stimulates a plant <NUM> or portion of a plant <NUM>, fertilizes or promotes growth (e.g., with a growth hormone) of a plant <NUM>, waters a plant <NUM>, applies light or some other radiation to a plant <NUM>, and/or injects one or more working fluids into the substrate <NUM> adjacent to a plant <NUM> (e.g., within a threshold distance from the plant). Other plant treatments are also possible. When applying a plant treatment, the treatment mechanisms <NUM> may be configured to spray one or more of: an herbicide, a fungicide, insecticide, some other pesticide, or water.

When the treatment is a substrate treatment, the treatment mechanism <NUM> applies a treatment to some portion of the substrate <NUM> in the field <NUM>. The treatment mechanism <NUM> may apply treatments to identified areas of the substrate <NUM>, or non-identified areas of the substrate <NUM>. For example, the farming machine <NUM> may identify and treat an area of substrate <NUM> in the field <NUM>. Alternatively, or additionally, the farming machine <NUM> may identify some other trigger that indicates a substrate <NUM> treatment and the treatment mechanism <NUM> may apply a treatment to the substrate <NUM>. Some example treatment mechanisms <NUM> configured for applying treatments to the substrate <NUM> include: one or more spray nozzles, one or more electromagnetic energy sources, one or more physical implements configured to manipulate the substrate <NUM>, but other substrate <NUM> treatment mechanisms <NUM> are also possible.

Of course, the farming machine <NUM> is not limited to treatment mechanisms <NUM> for plants <NUM> and substrates <NUM>. The farming machine <NUM> may include treatment mechanisms <NUM> for applying various other treatments to objects in the field <NUM>.

Depending on the configuration, the farming machine <NUM> may include various numbers of treatment mechanisms <NUM> (e.g., <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, etc.). A treatment mechanism <NUM> may be fixed (e.g., statically coupled) to the mounting mechanism <NUM> or attached to the farming machine <NUM>. Alternatively, or additionally, a treatment mechanism <NUM> may movable (e.g., translatable, rotatable, etc.) on the farming machine <NUM>. In one configuration, the farming machine <NUM> includes a single treatment mechanism <NUM>. In this case, the treatment mechanism <NUM> may be actuatable to align the treatment mechanism <NUM> to a treatment area <NUM>. In a second variation, the farming machine <NUM> includes a treatment mechanism <NUM> assembly comprising an array of treatment mechanisms <NUM>. In this configuration, a treatment mechanism <NUM> may be a single treatment mechanism <NUM>, a combination of treatment mechanisms <NUM>, or the treatment mechanism <NUM> assembly. Thus, either a single treatment mechanism <NUM>, a combination of treatment mechanisms <NUM>, or the entire assembly may be selected to apply a treatment to a treatment area <NUM>. Similarly, either the single, combination, or entire assembly may be actuated to align with a treatment area, as needed. In some configurations, the farming machine <NUM> may align a treatment mechanism <NUM> with an identified object in the operating environment <NUM>. That is, the farming machine <NUM> may identify an object in the operating environment <NUM> and actuate the treatment mechanism <NUM> such that its treatment area aligns with the identified object.

A treatment mechanism <NUM> may be operable between a standby mode and a treatment mode. In the standby mode the treatment mechanism <NUM> does not apply a treatment, and in the treatment mode the treatment mechanism <NUM> is controlled by the control system <NUM> to apply the treatment. However, the treatment mechanism <NUM> can be operable in any other suitable number of operation modes.

The configuration of the treatment mechanism <NUM> may affect parameters of the virtual safety bubble. For example, the treatment mechanism <NUM> may be collapsed in a compact configuration or deployed in an expanded configuration, and the control system generating the virtual safety bubble may automatically and/or dynamically adjust parameters of the safety bubble based on whether the treatment mechanism <NUM> is in the collapsed configuration or the expanded configuration. In another example, the treatment mechanism <NUM> may be operable for multiple farming actions. Based on the farming action, the control system may automatically and/or dynamically adjust parameters of the safety bubble.

The farming machine <NUM> includes a control system <NUM>. The control system <NUM> controls operation of the various components and systems on the farming machine <NUM>. For instance, the control system <NUM> can obtain information about the operating environment <NUM>, processes that information to identify a farming action to implement, and implement the identified farming action with system components of the farming machine <NUM>. The control system <NUM> may further aid in the navigation of the farming machine around the operating environment <NUM>. Navigation may include collecting and analyzing data relating to the environment from one or more sensors and generating navigation instructions based on the data.

The control system <NUM> can receive information from the detection mechanism <NUM>, the verification mechanism <NUM>, the treatment mechanism <NUM>, and/or any other component or system of the farming machine <NUM>. For example, the control system <NUM> may receive measurements from the detection mechanism <NUM> or verification mechanism <NUM>, or information relating to the state of a treatment mechanism <NUM> or implemented farming actions from a verification mechanism <NUM>. Other information is also possible.

Similarly, the control system <NUM> can provide input to the detection mechanism <NUM>, the verification mechanism <NUM>, and/or the treatment mechanism <NUM>. For instance, the control system <NUM> may be configured input and control operating parameters of the farming machine <NUM> (e.g., speed, direction). Similarly, the control system <NUM> may be configured to input and control operating parameters of the detection mechanism <NUM> and/or verification mechanism <NUM>. Operating parameters of the detection mechanism <NUM> and/or verification mechanism <NUM> may include processing time, location and/or angle of the detection mechanism <NUM>, image capture intervals, image capture settings, etc. Other inputs are also possible. Finally, the control system may be configured to generate machine inputs for the treatment mechanism <NUM>. That is translating a farming action of a treatment plan into machine instructions implementable by the treatment mechanism <NUM>.

The control system <NUM> can be operated by a user operating the farming machine <NUM>, wholly or partially autonomously, operated by a user connected to the farming machine <NUM> by a network, or any combination of the above. For instance, the control system <NUM> may be operated by an agricultural manager sitting in a cabin of the farming machine <NUM>, or the control system <NUM> may be operated by an agricultural manager connected to the control system <NUM> via a wireless network. In another example, the control system <NUM> may implement an array of control algorithms, machine vision algorithms, decision algorithms, etc. that allow it to operate autonomously or partially autonomously.

The control system <NUM> may be implemented by a computer or a system of distributed computers. The computers may be connected in various network environments. For example, the control system <NUM> may be a series of computers implemented on the farming machine <NUM> and connected by a local area network. In another example, the control system <NUM> may be a series of computers implemented on the farming machine <NUM>, in the cloud, a client device and connected by a wireless area network.

The control system <NUM> can apply one or more computer models to determine and implement farming actions in the field <NUM>. For example, the control system <NUM> can apply a plant identification module to images acquired by the detection mechanism <NUM> to determine and implement farming actions. The control system <NUM> may be coupled to the farming machine <NUM> such that an operator (e.g., a driver) can interact with the control system <NUM>. In other embodiments, the control system <NUM> is physically removed from the farming machine <NUM> and communicates with system components (e.g., detection mechanism <NUM>, treatment mechanism <NUM>, etc.) wirelessly.

In some configurations, the farming machine <NUM> may additionally include a communication apparatus, which functions to communicate (e.g., send and/or receive) data between the control system <NUM> and a set of remote devices. The communication apparatus can be a Wi-Fi communication system, a cellular communication system, a short-range communication system (e.g., Bluetooth, NFC, etc.), or any other suitable communication system.

In various configurations, the farming machine <NUM> may include any number of additional components.

For instance, the farming machine <NUM> may include a mounting mechanism <NUM>. The mounting mechanism <NUM> provides a mounting point for the components of the farming machine <NUM>. That is, the mounting mechanism <NUM> may be a chassis or frame to which components of the farming machine <NUM> may be attached but could alternatively be any other suitable mounting mechanism <NUM>. More generally, the mounting mechanism <NUM> statically retains and mechanically supports the positions of the detection mechanism <NUM>, the treatment mechanism <NUM>, and the verification mechanism <NUM>. In an example configuration, the mounting mechanism <NUM> extends outward from a body of the farming machine <NUM> such that the mounting mechanism <NUM> is approximately perpendicular to the direction of travel <NUM>. In some configurations, the mounting mechanism <NUM> may include an array of treatment mechanisms <NUM> positioned laterally along the mounting mechanism <NUM>. In some configurations, the farming machine <NUM> may not include a mounting mechanism <NUM>, the mounting mechanism <NUM> may be alternatively positioned, or the mounting mechanism <NUM> may be incorporated into any other component of the farming machine <NUM>.

The farming machine <NUM> may include locomoting mechanisms. The locomoting mechanisms may include any number of wheels, continuous treads, articulating legs, or some other locomoting mechanism(s). For instance, the farming machine <NUM> may include a first set and a second set of coaxial wheels, or a first set and a second set of continuous treads. In the either example, the rotational axis of the first and second set of wheels/treads are approximately parallel. Further, each set is arranged along opposing sides of the farming machine <NUM>. Typically, the locomoting mechanisms are attached to a drive mechanism that causes the locomoting mechanisms to translate the farming machine <NUM> through the operating environment <NUM>. For instance, the farming machine <NUM> may include a drive train for rotating wheels or treads. In different configurations, the farming machine <NUM> may include any other suitable number or combination of locomoting mechanisms and drive mechanisms.

The farming machine <NUM> may also include one or more coupling mechanisms <NUM> (e.g., a hitch). The coupling mechanism <NUM> functions to removably or statically couple various components of the farming machine <NUM>. For example, a coupling mechanism may attach a drive mechanism to a secondary component such that the secondary component is pulled behind the farming machine <NUM>. In another example, a coupling mechanism may couple one or more treatment mechanisms <NUM> to the farming machine <NUM>.

The farming machine <NUM> may additionally include a power source, which functions to power the system components, including the detection mechanism <NUM>, control system <NUM>, and treatment mechanism <NUM>. The power source can be mounted to the mounting mechanism <NUM>, can be removably coupled to the mounting mechanism <NUM>, or can be incorporated into another system component (e.g., located on the drive mechanism). The power source can be a rechargeable power source (e.g., a set of rechargeable batteries), an energy harvesting power source (e.g., a solar system), a fuel consuming power source (e.g., a set of fuel cells or an internal combustion system), or any other suitable power source. In other configurations, the power source can be incorporated into any other component of the farming machine <NUM>.

<FIG> is a block diagram of the system environment <NUM> for the farming machine <NUM>, in accordance with one or more embodiments. In this example, the control system <NUM> (e.g., control system <NUM>) is connected to external systems <NUM> and a machine component array <NUM> via a network <NUM> within the system environment <NUM>.

The external systems <NUM> are any system that can generate data representing information useful for determining and implementing farming actions in a field. External systems <NUM> may include one or more sensors <NUM>, one or more processing units <NUM>, and one or more datastores <NUM>. The one or more sensors <NUM> can measure the field <NUM>, the operating environment <NUM>, the farming machine <NUM>, etc. and generate data representing those measurements. For instance, the sensors <NUM> may include a rainfall sensor, a wind sensor, heat sensor, a camera, etc. The processing units <NUM> may process measured data to provide additional information that may aid in determining and implementing farming actions in the field. For instance, a processing unit <NUM> may access an image of a field <NUM> and calculate a weed pressure from the image or may access historical weather information for a field <NUM> to generate a forecast for the field. Datastores <NUM> store historical information regarding the farming machine <NUM>, the operating environment <NUM>, the field <NUM>, etc. that may be beneficial in determining and implementing farming actions in the field. For instance, the datastore <NUM> may store results of previously implemented treatment plans and farming actions for a field <NUM>, a nearby field, and or the region. The historical information may have been obtained from one or more farming machines (i.e., measuring the result of a farming action from a first farming machine with the sensors of a second farming machine). Further, the datastore <NUM> may store results of specific farming actions in the field <NUM>, or results of farming actions taken in nearby fields having similar characteristics. The datastore <NUM> may also store historical weather, flooding, field use, planted crops, etc. for the field and the surrounding area. Finally, the datastores <NUM> may store any information measured by other components in the system environment <NUM>.

The machine component array <NUM> includes one or more components <NUM>. Components <NUM> are elements of the farming machine <NUM> that can take farming actions (e.g., a treatment mechanism <NUM>). As illustrated, each component has one or more input controllers <NUM> and one or more sensors <NUM>, but a component may include only sensors <NUM> or only input controllers <NUM>. An input controller <NUM> controls the function of the component <NUM>. For example, an input controller <NUM> may receive machine commands via the network <NUM> and actuate the component <NUM> in response. A sensor <NUM> generates data representing measurements of the operating environment <NUM> and provides that data to other systems and components within the system environment <NUM>. The measurements may be of a component <NUM>, the farming machine <NUM>, the operating environment <NUM>, etc. For example, a sensor <NUM> may measure a configuration or state of the component <NUM> (e.g., a setting, parameter, power load, etc.), measure conditions in the operating environment <NUM> (e.g., moisture, temperature, etc.), capture information representing the operating environment <NUM> (e.g., images, depth information, distance information), and generate data representing the measurement(s).

The control system <NUM> receives information from external systems <NUM> and the machine component array <NUM> and implements a treatment plan in a field using a farming machine <NUM>. Before implementing the treatment plan, the farming machine verifies that it is safe to operate. To do so, the control system <NUM> receives a notification from a manager that the environment surrounding the farming machine is safe for operation and empty of obstacles. The control system <NUM> verifies, using captured images, that there are no obstacles in the environment surrounding the farming machine. The control system <NUM> generates a virtual safety bubble for the farming actions based on a configuration of the farming machine. While the farming machine is implementing the farming actions, the control system <NUM> continually identifies and locate obstacles in the environment. If one of the obstacles is within the virtual safety bubble, the control system <NUM> may stop operation or enact preventive measures.

The control system <NUM> includes a safety bubble generation module <NUM>, a classification module <NUM>, a safety module <NUM>, a navigation module <NUM>, and a user interface module <NUM>. In other embodiments, the control system <NUM> has additional/fewer modules. In other embodiments, the modules may be variably configured such that functions of one may be performable by one or more other modules.

The safety bubble generation module <NUM> generates a virtual safety bubble for the farming machine <NUM>. The virtual safety bubble may be a three-dimensional shape around the farming machine <NUM>. In other embodiments, the virtual safety bubble may have a belt shape, e.g., a wall of certain height that surrounds the farming machine <NUM>. Various other shapes and sizes may be envisioned. The safety bubble generation module <NUM> sets the shape and size of the virtual safety bubble based on the configuration of the farming machine <NUM>. For example, the safety bubble generation module <NUM> may determine a shape and/or a size of the virtual safety bubble based on whether the farming machine <NUM> is in a first configuration for navigating to an operating environment or in a second configuration for performing a treatment plan. The safety bubble generation module <NUM> may dynamically adjust the virtual safety bubble based on sensor data. For example, the safety bubble generation module <NUM> may increase the virtual safety bubble size in response to a sunset darkening the operating environment.

The classification module <NUM> classifies objects in the images captured by the cameras (embodiment of sensors <NUM>) implemented on the farming machine100. The classification module <NUM> may utilize one or more models to classify pixels relating to objects the image. One model may identify obstacles as objects not part of the farming operation. For example, the model may classify rows of crop as non-obstacles but would classify a wild fox or a large boulder as an obstacle. Another model may perform image segmentation, classifying pixels for various object types, e.g., the ground, the sky, foliage, obstacles, etc. Yet another model may calculate a velocity of objects relative to the farming machine <NUM>, e.g., using one or more visual odometry methods. And still another model may predict depth of the objects from the camera, e.g., utilizing a depth estimation model trained to predict the depth based on image data. Depth generally refers to the distance between the farming machine and pixels or objects in the images. For example, a first object present in an image can be determined to be at a depth of <NUM> meters from the farming machine. The classification module <NUM> may further generate 3D point cloud representations of objects within a virtual operating environment, allowing for tracking of objects. The various models may input other sensor data (captured by the sensors <NUM> or the sensors <NUM>) to aid in the classification, e.g., LIDAR data, temperature measurements, etc..

The safety module <NUM> evaluates whether obstacles are within the virtual safety bubble. The safety module <NUM> may utilize a depth estimation model to predict depths of obstacles relative to the farming machine <NUM>. If an obstacle has a depth that is below the virtual safety below, i.e., some portion of the obstacle breaks the barrier of virtual safety bubble, then the safety module <NUM> provides that notice to the navigation module <NUM>, e.g., for ceasing operation or enacting preventive measures.

The navigation module <NUM> navigates the farming machine <NUM>. The navigation module <NUM> generates navigation instructions based on a treatment plan. The treatment plan may include one or more farming operations to be completed. The navigation module <NUM> may chart a route to navigate the vehicle. The navigation module <NUM> may adjust the navigation route based on sensor data. The navigation module <NUM> may receive notices from the safety module <NUM> that an obstacle has breached the virtual safety bubble. In response to the notice, the navigation module <NUM> may cease operations, enact other preventive measures, or some combination thereof. In one example of a prevent measure, the navigation module <NUM> can bring the farming machine <NUM> to a stop when notice is given that an obstacle has breached the virtual safety bubble. As another example of a prevent measure, the navigation module <NUM> can chart a route around the obstacle to prevent collision. Additional details relating to navigation by the navigation module <NUM> is described in <FIG>.

The user interface module <NUM> maintains a graphical user interface (GUI) for displaying information to the manager of the farming machine <NUM> and receiving inputs from the manager. The user interface module <NUM> may graphically illustrate the farming machine <NUM> in operation, e.g., when moving along a path, or when performing one or more farming actions. The GUI may also display any obstacles or other objects in the operating environment. The GUI may further be configured to receive inputs to control the farming machine <NUM>. Example inputs include toggling a speed to the farming machine <NUM>, manual adjustment of the virtual safety bubble, etc. In one embodiment, the GUI may notify a manager of the farming machine <NUM> that an obstacle has breached the virtual safety bubble, the GUI may request action or input from the manage in how to respond. Example user interfaces are further described in <FIG>.

In one or more embodiments, the models used by the control system <NUM> may be trained as machine-learning models using training data. The training may be supervised, unsupervised, or semi-supervised. Various types of machine-learning model architectures may be implemented, e.g., neural networks, decision trees, support vector machine learning, etc..

The network <NUM> connects nodes of the system environment <NUM> to allow microcontrollers and devices to communicate with each other. In some embodiments, the components are connected within the network as a Controller Area Network (CAN). In this case, within the network each element has an input and output connection, and the network <NUM> can translate information between the various elements. For example, the network <NUM> receives input information from the camera array <NUM> and component array <NUM>, processes the information, and transmits the information to the control system <NUM>. The control system <NUM> generates a farming action based on the information and transmits instructions to implement the farming action to the appropriate component(s) <NUM> of the component array <NUM>.

Additionally, the system environment <NUM> may be other types of network environments and include other networks, or a combination of network environments with several networks. For example, the system environment <NUM>, can be a network such as the Internet, a LAN, a MAN, a WAN, a mobile wired or wireless network, a private network, a virtual private network, a direct communication line, and the like.

As described above, a farming machine is configured with one or more detection mechanisms ("detection system") to measure the environment. In one configuration the detection system may be an array of detection mechanisms configured to capture images of the environment. Image data in the image represent the various objects in the environment surrounding the farming machine. Thus, the detection system is configured to capture image data of the environment.

The detection system has a field of view, and because the detection system is an array of detection mechanisms, the detection system's field of view may comprise of several fields of view that may be composited together to form a <NUM>-degree view. That is, each detection mechanism has its own field of view, and the fields of view, in aggregate, form the field of view of the detection system.

There may be one or more blind spots in a field of view caused by the configuration of the detection system. Some blind spots can include areas outside of reach of any detection mechanism and obstructed views, e.g., views within the field of view of the detection system but obstructed by one or more objects. Obstructed views comprise image data in images where an object obstructs an object or objects behind it (such that obstructed objects are obscured from view). Unobstructed views comprise image data in images where no objects obstruct an object or objects behind it. For example, consider a detection mechanism capturing images of a tire coupled to the farming machine and the surrounding environment. Because the tire is obscuring image data of objects behind the tire (e.g., ground, rocks, etc.) it is an obstructed view. The remainder of the image is an unobstructed view because there are no objects obscuring other objects.

Obstructed views are problematic in autonomous farming due to their inherent safety issues. For example, an object that may be a significant obstacle may be obscured by another object in an obstructed view. The farming machine may therefore be unable to identify and account for a problematic obstacle. Methods are presented herein to establish a virtual safety bubble to prevent obstacles from entering obstructed views of the farming machine.

<FIG> illustrates a farming machine <NUM> (an embodiment of the farming machine <NUM>) pulling an implement <NUM> and outfitted with a detection system. The farming machine <NUM> has a detection system with a total of six detection mechanisms <NUM>. Three detection mechanisms 310A, 310B, and 310C are positioned on a front end of the farming machine <NUM>, with the remaining three detection mechanisms 310D, 310E, 310F positioned on a back end of the farming machine <NUM> towards the implement <NUM>. As noted above the detection system's field of view may aggregate the individual fields of view <NUM> from the detection mechanisms. Detection mechanism 310A has field of view 315A. Detection mechanism 310B has field of view 315B. Detection mechanism 310C has field of view 315C. Detection mechanism 310D has field of view 315D. Detection mechanism 310E has field of view 315E. Detection mechanism 310F has field of view 315F.

<FIG> also illustrates two obstacles. The first obstacle <NUM> is just off the front-right tire of the farming machine. The second obstacle <NUM> is to the front right of the farming machine. The farming machine <NUM> is configured to apply an obstacle detection model to images captured by the detection mechanism to identify the obstacles in the environment. That is, the farming machine <NUM> employs the obstacle detection model to determine that pixels in images represent obstacles and locates the approximate location of those obstacles in the environment (e.g., by estimating depth).

As described above, the detection system of the farming machine <NUM> includes various blind spots. Blind spots are areas in the environment not visible by the farming machine because, for instance, a portion of the farming machine obstructs the view (e.g., behind a tire), or the detection mechanisms are nor positioned to capture that portion of the environment (e.g., under the tractor).

<FIG> illustrates one or more blind spots of the farming machine in <FIG>. The blind spots are indicated by polygons. The first blind spot 340A may be a combination of the footprint of the farming machine <NUM> (including vehicle frame, tires, other components obstructing views) and gaps between detection mechanisms (e.g., a gap on the left between detection mechanisms 310B and 310E and another gap on the right between detection mechanisms 310C and 310F). The second blind spot 340B may be an obstructed view caused by the farming implement <NUM> having some height blocking views.

<FIG> also illustrates the two obstacles. Here, the first obstacle <NUM> is partially in the blind spot 340A and the second obstacle <NUM> is in unobstructed view and within the field of view of the detection system. As such, when applying the obstacle detection model to images captured by the detection system, farming machine would not be able to identify and locate the first obstacle <NUM> but would be able to identify and locate the second obstacle <NUM>.

The farming machine may be configured to only begin autonomously implementing farming actions if a manager of the farming machine verifies the environment. That is, a manager of the farming machine must verify that there are no obstacles in obstructed and/or unobstructed views of the farming machine. In essence, the manager must walk around the farming machine to verify that there are no obstacles in areas undetectable by the detection system. In some configurations, the verification process may include playing sirens and flashing lights to make it apparent that the farming machine is about to begin autonomously farming. The lights and sirens make it more likely that any humans in the environment will exit the environment.

As part of the verification process, the farming machine may communicate with a control system operated by the manager. That is, the farming machine may transmit and receive information from a control system operated by a manager. For example, the farming machine may transmit a request for the manager to verify the environment, and the farming machine may receive a verification of the environment in response (once the manager verifies the environment).

The farming machine includes a virtual safety bubble generation module configured to generate a virtual safety bubble. A virtual safety bubble is an area in the environment which enables the farming machine to operate autonomously without colliding with obstacles. A virtual safety bubble may be an area in the environment (<NUM>) directly surrounding the farming machine, (<NUM>) in a forward path of the farming machine, (<NUM>) in a backward path from the farming machine, (<NUM>) along an expected path of the farming machine, and/or some area in the environment.

The farming machine generates the virtual safety bubble based on the configuration of the farming machine. Here, "configuration" is a term used to describe several aspects of the farming machine, implement, and environment which can be used to generate the virtual safety bubble. A non-exhaustive list of aspects of the farming machine configuration that may affect the virtual safety bubble follows.

Machine Path. The machine path may describe a current path of a machine or an expected path of the machine. The machine path may be in any direction relative to the current position of the farming machine. Additionally, the virtual safety bubble for the machine path may consider machine characteristics of the farming machine. , the virtual safety bubble for a large farming machine along its machine path is larger than that of a smaller farming machine.

Velocity may be a current or scheduled velocity of the farming machine. As implemented by the farming machine, velocity may be a scalar or a vector.

Acceleration. Acceleration be a current or scheduled accretion of the farming machine. As implemented by the farming machine, acceleration may be a scalar or a vector.

Expected Obstacle Characteristics. Expected obstacle characteristics are characteristics of obstacles a farming machine may expect to find in its environment. For instance, a farming machine operating near building may expect to find different obstacles than one operating in a field. As such, each environment may have correspondingly different virtual safety bubbles.

Implement Type. Implement type is the type of implement being employed by the farming machine (if any). As an example, an implement may be a tiller, a seeder, a sprayer, etc. Accordingly, each implement may indicate parameters for their virtual safety bubbles.

Mounting Mechanism Type. Mounting mechanism type describes how various parts of the farming machine are attached to the structure. For instance, a mounting mechanism may be a hitch, and the hitch may be a mobile hitch or a static hitch. Accordingly, the type of mounting mechanism may indicate parameters for the virtual safety bubble.

Type of Farming Actions. Farming actions are described in detail above. Different farming actions may indicate different parameters for the virtual safety bubble. For instance, a virtual safety bubble for spraying weeds may be different than a virtual safety bubble for tilling a field. The farming machine's control system may determine a direction that a farming action would face to aid in determination of the parameters of the virtual safety bubble (e.g., the shape and the size of the virtual safety bubble). For example, the control system can set a shape of the virtual safety bubble to be predominantly in front of the farming machine based on the farming action being tilling. In another example, the control system can set a shape of the virtual safety bubble with a radius around the treatment mechanism in a spraying mode. The control system may also access a physical configuration of the farming machine based on the farming action being performed, e.g., a first farming action may place the farming machine in a first physical configuration, whereas a second farming action may place the farming machine in a second physical configuration that is different than the first physical configuration.

Implementation Characteristics for Farming Actions. Implementation characteristics describes the particulars of how a farming machine implements a farming action. Some characteristics may include, for example, a spray speed of a spray nozzle, a head height of a harvester, etc..

Machine Characteristics for Farming Machine. Machine characteristics describe the physical manifestation of the farming machine. That is, the size, shape, and spatial characteristics of the farming machine. The farming machine may store a digital representation of its machine characteristics that may be accessed when generating a virtual safety bubble.

Implement Characteristics for Implement. Implement characteristics describe the physical manifestation of the farming implement. That is, the size, shape, and spatial characteristics of the farming implement. The farming implement may store a digital representation of its implement characteristics that may be accessed when generating a virtual safety bubble.

Characteristics of Other Attachments. Other attachments may include any one component that is attached to the farming machine or implement. For example, the farming machine can be rigged with additional flood lights which may expand the dimensional profile of the farming machine.

Environment Characteristics. Environment characteristics describes the working environment of the farming machine. Some example environment characteristics include the size, shape, and spatial characteristics of the field in which the farming machine operates. Environment characteristics may also describe the weather.

Obstacle Type. Obstacles may be dynamic (i.e., moving) or static (i.e., unmoving). The farming machine may generate a different virtual safety bubble for an identified dynamic and/or static obstacle.

Manager Input. Manager input is information from the manager that may be used to generate a virtual safety bubble. Manager input may include any of the aforementioned configuration information.

Local Regulations. The control system can maintain a log of different local regulations depending on a geographical location of the farming machine. In one or more examples, a first country may have different regulations than a second country; a first state may have different regulations than a second state; a first city may have different regulations than a second city; or some combination thereof. The different regulations can limit the farming actions, e.g., speed limit, permitted period of operation, permitted weather for operation, other regulations, etc..

To refresh, the farming machine utilizes a machine configuration to determine a virtual safety bubble around the farming machine. The machine configuration may be any of the aforementioned configuration information. The virtual safety bubble may be represented as a relative distance, an absolute distance, a depth, a time (e.g., based on velocity and/or acceleration), legal requirements, or any other suitable metric for quantifying the virtual safety bubble.

The farming machine continually monitors the environment such that no obstacles are within the virtual safety bubble. That is, the detection mechanisms capture images, the farming machine applies an obstacle identification model to the images and identifies and locates an obstacle in the environment. If the farming machine identifies an obstacle in the virtual safety bubble, it terminates operation of the farming machine.

Notably, the farming machine may treat obstacles and objects in different manners. For instance, a farming machine may identify a large pile of leaves in a virtual safety bubble, identify it as an object, and continue performing farming actions because the leaves would not damage the farming machine on contact. To the contrary, a farming machine may identify a log in a virtual safety bubble, identify it as an object, classify it as an obstacle, and cease performing farming actions because the log would damage the farming machine in a collision.

In some examples, the farming machine may treat different types of obstacles in different manners. For instance, a dynamic obstacle (e.g., a human, a moving car, etc.) may warrant different virtual safety bubbles relative to a static obstacle (e.g., a log, a chair, etc.). Naturally, dynamic obstacles likely indicate larger virtual safety bubbles because of their ability to move through the environment, while static obstacles likely indicate smaller virtual safety bubbles because they remain stationary. In some examples, the farming machine may treat humans in a different manner than all other obstacles. For instance, the farming machine may generate a virtual safety bubble for humans that is larger than all other objects and obstacles. In one or more embodiments, the farming machine may generate a plurality of virtual safety bubbles utilized concurrently. A first virtual safety bubble may be defined for a first class of objects (e.g., humans), and a second virtual safety bubble may be defined for a second class of objects (e.g., obstacles).

<FIG> and <FIG> illustrate dynamically modifying the virtual safety bubble around a farming machine. In <FIG> & <FIG>, the farming machine is similarly configured to the farming machine in <FIG>, having a detection system comprising at least six detection mechanisms compositing fields of view to create the <NUM>-degree field of view all around the farming machine.

<FIG> illustrates a first virtual safety bubble <NUM> around a farming machine, in accordance with one or more embodiments. The farming machine is autonomously implementing farming actions in the environment (e.g., tilling) using a particular configuration. The configuration of the farming machine describes its spatial characteristics (e.g., positions of tires, implements, etc.), the farming actions to implement, and the implementation characteristics for those farming actions (e.g., type, path, velocity, acceleration, implementation settings, etc.).

Based on the configuration, the farming machine determines a virtual safety bubble <NUM>. The virtual safety bubble <NUM> is represented by the oval surrounding the farming machine. The virtual safety bubble <NUM> represents a safe operational area where there are no identified obstacles. To maintain the virtual safety bubble <NUM>, the farming machine continuously captures images and applies an obstacle detection model to locate obstacles in the environment. If the farming machine detects an obstacle <NUM> in the virtual safety bubble <NUM> the farming machine may terminate operation. That is, the farming machine ceases implementation of farming actions and may cease movement. Implementing the virtual safety bubble beyond the blind spots prevents any obstacle from being obscured and missed by the detection system, which could cause a collision and damage to the farming machine, object, or individual.

<FIG> illustrates dynamically modifying the virtual safety bubble around a farming machine, in accordance with one or more embodiments. To illustrate, imagine that the farming actions implemented by the farming machine in <FIG> are performed with a first configuration with the virtual safety bubble <NUM>. For example, the farming machine is performing farming actions at a first velocity in the environment. Now consider, for example, the farming machine changes its configuration. For example, the manager instructs the farming machine to travel more quickly in the environment as it implements farming actions. As another example, the farming machine enters an environment populated with cattle or other animals. The farming machine may change from the first configuration to a second configuration, creating a larger virtual safety bubble.

The farming machine calculates a new virtual safety bubble <NUM> to account for the second configuration. The new virtual safety bubble <NUM> is illustrated by the dashed oval in <FIG>. Notably, the new virtual safety bubble <NUM> is larger than the original virtual safety bubble, e.g., to account for the farming machine's higher velocity. This larger virtual safety bubble <NUM> may be generated for several reasons, one of which is to allow for the greater time required for implementing a corrective action when an obstacle is detected due to a higher speed.

To illustrate, recall the obstacle <NUM> in <FIG>. There, the obstacle <NUM> was outside the virtual safety bubble <NUM> and the farming machine would not cease operations. 4B the farming machine is moving faster. If the new virtual safety bubble <NUM> was the same size as the original virtual safety bubble <NUM>, the farming machine may collide with the obstacle <NUM> before it can take corrective action. However, because the virtual safety bubble <NUM> is larger than the original virtual safety bubble <NUM>, the farming machine identifies the obstacle <NUM> and may take corrective action before colliding with the obstacle <NUM>.

The farming machine may be configured to generate a virtual safety bubble around the farming machine that allows for safe, autonomous implementation of farming actions. <FIG> illustrates a process flow for generating a virtual safety bubble, according to one example embodiment. Although <FIG> is described from the perspective of the farming machine, any component of the farming machine may perform one or more of the steps (e.g., the control system <NUM> or <NUM>). In other embodiments, there may be additional or fewer steps. In other embodiments, the steps listed may occur in a different order.

To provide context, an autonomous farming machine is configured with a detection system. The detection system may comprise six cameras positioned around the farming machine that provide the farming machine a <NUM>-degree field view of the environment. Within the field of view are obstructed views and unobstructed views. Obstructed views are image data within the field of view where an object in the environment obscures portions of the environment behind the object from the detection mechanism (e.g., behind a tire, or under the cab). Unobstructed views are image data within the field of view that are not obstructed.

The farming machine receives a notification to begin autonomously implementing farming actions in the environment. In response, the farming machine transmits a request to verify that the operating environment of the farming machine is safe. Verification may include transmitting a notification to the manager to verify that there are no obstacles in the obstructed views of the detection system. The manager verifies that there are no obstacles and transmits a notification to the farming machine reflecting the verification.

The farming machine receives <NUM> a notification that there are no obstacles in the blind spots of the detection system. The manager may provide such notification, e.g., via a GUI running on a mobile phone application.

The farming machine verifies <NUM> that there are no obstacles in the unobstructed views of the environment using an obstacle detection model. That is, the farming machine captures one or more images of the environment using the detection system and applies an obstacle detection model to the images. The obstacle detection model analyzes the images to determine whether any of the pixels in the image represent an obstacle.

The farming machine receives <NUM> instructions from the farmer to begin autonomously performing farming actions in the field. In an example configuration, the farming machine may be unable to begin autonomous performance without a verification from the manager that there are no obstacles in the obstructed views and verifying (itself) that there are no obstacles in the unobstructed views.

The farming machine determines <NUM> a configuration of the farming machine to perform the prescribed farming actions in the environment. Determining the configuration may include accessing an implement capability, a computer model of the farming machine, types of farming actions, and implementation characteristics defining how the farming machine implements the farming actions (e.g., speed, path, etc.).

The farming machine determines <NUM> a virtual safety bubble based on the determined configuration. The virtual safety bubble represents an area surrounding the farming machine where, if an obstacle is detected in the area, the farming machine will cease operation. The virtual safety bubble may be a distance, a time, a depth, a relative position, or any other measure of a virtual safety bubble.

The farming machine detects <NUM> an obstacle in the environment based on applying the obstacle detection model to the images captured by the detection system. As the farming machine performs farming actions in the field the detection mechanism continuously captures images of the environment. Moreover, the farming machine continuously applies the obstacle detection model to the captured images to identify obstacles in the environment.

The farming machine determines <NUM> that an obstacle is within the virtual safety bubble. The farming machine may determine that the obstacle has breached the virtual safety bubble if a depth of the obstacle is at or below the virtual safety bubble. The depth may be determined via a detection and ranging sensor, or a depth estimation model applied to the images.

In response to determining that an obstacle is in the virtual safety bubble, the farming machine terminates <NUM> operation. That is, the farming machine stops implementing the farming actions in the field. In other embodiments, the farming machine may enact other preventive measures in response to detecting an obstacle having breached the virtual safety bubble.

As described above the farming machine may interact with a manager when performing farming actions in the field. Some of these interactions may be keyed to when the farming machine detects an object in its virtual safety bubble. Once detected, the farming machine may transmit to, or receive information from, a manager of the farming machine. The farming machine may also transmit and receive information when establishing a virtual safety bubble around the farming machine. <FIG> illustrate various examples of a client device interacting with a farming machine.

<FIG> illustrates a verification process of the detection systems of the farming machine. The verification process may include verifying that there are no obstacles visible in obstructed views of the farming machine. On the left panel, the GUI illustrates the farming machine and implement with six zones where the cameras are positioned and directed. The GUI prompts the manager to "walk around the machine to validate the cameras. " As the manager physically walks around the farming machine, each of the detection mechanisms (e.g., cameras) may capture data that is used by the farming machine to validate the detection mechanisms' ability to detect the manager. The right panel shows a completed walk-around with checkmarks next to each detection mechanism (e.g., camera).

<FIG> illustrates a notification that the farming machine is establishing the virtual safety bubble. That is, once implemented, the virtual safety bubble will be maintained according to the methods described above. So, if a human or object enters the virtual safety bubble the farming machine may take corresponding actions as outlined above. The left panel shows a slider <NUM> that allows for a manager to engage the farming machine in the farming actions. Sliding the slider <NUM> to the right is an embodiment of step <NUM> in <FIG> of providing and receiving instructions to begin autonomously performing farming actions.

<FIG> illustrates a notification transmitted to a client device regarding a detected obstacle. The notification may occur when the object is detected within the virtual safety bubble. The notification may include characteristics describing the detected object. On the left panel, an obstacle notification <NUM> is shown as a pop-up notification on a mobile device. Upon receiving a click from the manager, the mobile application can expand to provide a detailed obstacle report <NUM>, shown in the middle panel, providing additional details on the detected obstacle. The detailed obstacle report <NUM> may include an option to access an obstacle video feed <NUM> captured by a detection mechanism, shown in the right panel. The detailed obstacle report <NUM> can further include preventive measures that can be undertaken by the farming machine.

<FIG> illustrates actions the farming machine may implement when detecting an object in the virtual safety bubble. For example, the farming machine may route around the object in the field. The GUI can illustrate a route around the obstacle and progress of the farming machine in navigating the route, shown in the left panel. Upon completion of the route, the GUI can notify the manager of successful routing around the obstacle, shown in the middle panel. The right panel illustrates another example screenshot that illustrates an alternative route around an obstacle with an actionable option to instruct the farming machine to enact the prevent measure of navigating around the obstacle.

<FIG> illustrates actions the farming machine may implement when detecting an object in the virtual safety bubble. For example, the farming machine may cease operation in the field. In the left panel, the GUI illustrates that the farming machine has ceased operations (paused) in response to detection of an obstacle. In the middle panel, the GUI indicates the farming machine is shutting down after being "idle for <NUM> minutes" after having paused due to detecting the obstacle. In the right panel, the GUI indicates that the farming machine is "shutting down," e.g., switching to an inactive state.

<FIG> illustrates a navigational workflow <NUM> of farming machine, in accordance with one or more embodiments. The farming machine may implement the control system <NUM> as described in <FIG>. In other embodiments, the navigational workflow <NUM> may include additional steps, fewer steps, steps in a different order, or some combination thereof. Although the following description is in the perspective of the control system <NUM>, the farming machine at large may also perform the navigational workflow (e.g., via distributed systems in contrast to one control system).

The control system <NUM> begins by detecting objects in an operating environment of the farming machine. The control system <NUM> utilizes a spatial engine <NUM> that generates an occupancy grid <NUM>. The occupancy grid <NUM> is a virtual representation of the spatial environment of the farming machine. The control system <NUM> may further utilize a route engine <NUM> that generates an active path <NUM> for the farming machine to travel on. The controls system <NUM> may further receive GPS coordinates <NUM>, e.g., from a GPS receiver. The control system <NUM> performs passive mapping <NUM>, detecting objects <NUM> in the environment of the farming machine. The control system <NUM> performs object tracking <NUM>, e.g., by constantly updating a position of an object relative to the farming machine within the occupancy grid <NUM>.

In one or more embodiments, the control system <NUM> may utilize object tracking <NUM> to determine whether an object may have entered a blind spot. The control system <NUM> may track an object present in a plurality of images. Upon determining that the object has disappeared from view, i.e., no longer present in any of the images, the control system <NUM> may determine the object to have entered a blind spot. In other embodiments, the control system <NUM>, knowing that an object is likely in a blind spot, may prompt a user to verify whether the object has been cleared or remains in the blind spot. In response to the user providing an input indicating the object has been cleared, then the control system <NUM> may continue <NUM> operation. In response to the user providing an input indicating that the object remains in the blind spot, the control system <NUM> may reroute. The control system <NUM> may request further input from the manager via step <NUM>.

The control system <NUM> detects an obstacle on the active path <NUM>. As noted, the control system <NUM> may utilize a virtual safety bubble to detect when obstacles have breached the virtual safety bubble. In response to detecting the obstacle has breached the virtual safety bubble, the controls system <NUM> stops <NUM> operations (or enact other preventive measures). The control system <NUM> notifies <NUM> the manager of the obstacle in path (e.g., as shown in <FIG>). The control system <NUM> receives <NUM> input from the manager, e.g., to approve <NUM> of the object, i.e., to override object as not an obstacle, allowing for continued operation <NUM>. Otherwise, the manager may provide input to reroute <NUM>. In response, the control system <NUM> may reroute path <NUM> around the obstacle. Once cleared, the controls system <NUM> can continue <NUM> farming actions.

In one or more embodiments, the control system <NUM> can routinely update bounding boxes of the objects. The control system <NUM> can routinely evaluate whether a bounding box for an object is accurately defined for the object. If not accurately defined, the control system <NUM> may implement Spark AI <NUM> to produce corrected bounding boxes <NUM> for the various objects. Having accurate bounding boxes increases detect precision, i.e., when detecting the object breaches the virtual safety bubble.

<FIG> illustrates navigation of a farming machine <NUM> on-path on a straight path <NUM>, in accordance with one or more embodiments. The farming machine <NUM> is an embodiment of the farming machine <NUM> comprising the control system <NUM>. The farming machine <NUM> generates the virtual safety bubble <NUM> to aid navigation of the farming machine <NUM>. As the farming machine <NUM> is driving on the path <NUM> and encounters an obstacle <NUM> (i.e., the obstacle <NUM> breaches the virtual safety bubble <NUM>), then the farming machine <NUM> may cease operations and/or enact other preventive measures.

<FIG> illustrates navigation of a farming machine <NUM> off-path on a straight path <NUM>, in accordance with one or more embodiments. The farming machine <NUM> is an embodiment of the farming machine <NUM> comprising the control system <NUM>. The farming machine <NUM> generates the virtual safety bubble <NUM> to aid navigation of the farming machine <NUM>. In this scenario, the farming machine <NUM> is significantly off-path. If the farming machine <NUM> determines that it is off-path, then the farming machine <NUM> may generate course-correction navigation instructions to route the farming machine <NUM> back onto the path <NUM>. The farming machine <NUM> may also cease operations and/or provide a notification to a manager indicating that the farming machine <NUM> is off-path, requesting subsequent instructions. Even when off-path, if the farming machine <NUM> encounters an obstacle <NUM> (i.e., the obstacle <NUM> breaches the virtual safety bubble <NUM>), then the farming machine <NUM> may cease operations and/or enact other preventive measures.

<FIG> illustrates navigation of a farming machine <NUM> on-path and off-center of a straight path <NUM>, in accordance with one or more embodiments. The farming machine <NUM> is an embodiment of the farming machine <NUM> comprising the control system <NUM>. The farming machine <NUM> generates the virtual safety bubble <NUM> to aid navigation of the farming machine <NUM>. In this scenario, the farming machine <NUM> is on-path but off-center. If the farming machine <NUM> determines that it is off-center, then the farming machine <NUM> may generate course-correction navigation instructions to route the farming machine <NUM> back onto the center of the path <NUM>. Even when off-path, if the farming machine <NUM> encounters an obstacle <NUM> (i.e., the obstacle <NUM> breaches the virtual safety bubble <NUM>), then the farming machine <NUM> may cease operations and/or enact other preventive measures.

<FIG> illustrates navigation of a farming machine <NUM> when off-path but perceived to be on-path, in accordance with one or more embodiments. The farming machine <NUM> is an embodiment of the farming machine <NUM> comprising the control system <NUM>. The farming machine <NUM> may receive GPS coordinates such that a perceived position <NUM> of the farming machine <NUM> is on-path, i.e., on the path <NUM>. However, in fact, the farming machine <NUM> is off-path. The farming machine <NUM> utilizes the virtual safety bubble <NUM>, but will only enact preventive measures when the obstacle <NUM> (which is off-path) enters the virtual safety bubble <NUM>. Obstacles that are on the actual path <NUM> may not breach the virtual safety bubble <NUM>, such that the farming machine will continue operations. In some embodiments, the farming machine <NUM> may received corrected GPS coordinates locating the farming machine <NUM> off-path, although previously perceived to be on-path, at which point, the farming machine <NUM> may generate and enact course-correction navigation to navigate the farming machine <NUM> back onto the path <NUM>.

<FIG> illustrates navigation of a farming machine <NUM> when on-path but perceived to be off-path, in accordance with one or more embodiments. This scenario is a converse to the scenario in <FIG>. If the farming machine <NUM> encounters obstacle <NUM> on the path <NUM>, though perceived to be off-path, e.g., perceived obstacle <NUM> is not on the path <NUM>, the farming machine <NUM> will enact preventive measures.

<FIG> illustrates navigation of a farming machine <NUM> when on-turn on a curved path <NUM>, in accordance with one or more embodiments. On-turn refers to the control system's perceived turning curvature matching to the target turning curvature to remain on the curved path <NUM> when performing the turn. Off-turn refers to the control system's perceived turning curvature rotationally offset from the target turning curvature to remain on the curved path <NUM>. The farming machine <NUM> is an embodiment of the farming machine <NUM> comprising the control system <NUM>. When on a curved path <NUM>, the farming machine <NUM> may adjust the virtual safety bubble <NUM> to account for the turning radius of the farming machine <NUM>. For example, the virtual safety bubble <NUM> may be extended in a turning direction of the farming machine <NUM>. When the farming machine <NUM> detects one or more obstacles <NUM> and <NUM> are within the virtual safety bubble <NUM>, the farming machine <NUM> can enact preventive measures.

<FIG> illustrates navigation of a farming machine when off-turn on a curved path <NUM>, in accordance with one or more embodiments. The farming machine <NUM> is an embodiment of the farming machine <NUM> comprising the control system <NUM>. The farming machine <NUM> may have a perceived orientation that is skewed from the actual orientation. In such scenario, the farming machine <NUM> is traveling along a perceived curved path <NUM> that is offset from the curved path <NUM>. The farming machine <NUM> may enact course-correction navigation to align the farming machine's <NUM> orientation, i.e., to align the perceived path <NUM> to the actual path <NUM>. In one or more embodiments, the farming machine <NUM> can utilize the detection mechanisms to locate obstacles <NUM> and <NUM> on the path <NUM> as markers on the path <NUM>.

<FIG> is a block diagram illustrating components of an example machine able to read instructions from a machine-readable medium and execute them in a processor (or controller). Specifically, <FIG> shows a diagrammatic representation of a machine in the example form a computer system <NUM>, within which program code (e.g., software or software modules) for causing the machine to perform any one or more of the methodologies discussed herein may be executed. The program code may be comprised of instructions <NUM> executable by one or more processors <NUM>. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions <NUM> (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute instructions <NUM> to perform any one or more of the methodologies discussed herein.

The example computer system <NUM> includes a processor <NUM> (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination of these), a main memory <NUM>, and a static memory <NUM>, which are configured to communicate with each other via a bus <NUM>. The computer system <NUM> may further include visual display interface <NUM>. The visual interface may include a software driver that enables displaying user interfaces on a screen (or display). The visual interface may display user interfaces directly (e.g., on the screen) or indirectly on a surface, window, or the like (e.g., via a visual projection unit). For ease of discussion the visual interface may be described as a screen. The visual interface <NUM> may include or may interface with a touch enabled screen. The computer system <NUM> may also include alphanumeric input device <NUM> (e.g., a keyboard or touch screen keyboard), a cursor control device <NUM> (e.g., a mouse, a trackball, a joystick, a motion sensor, or other pointing instrument), a storage unit <NUM>, a signal generation device <NUM> (e.g., a speaker), and a network interface device <NUM>, which also are configured to communicate via the bus <NUM>.

The storage unit <NUM> includes a machine-readable medium <NUM> on which is stored instructions <NUM> (e.g., software) embodying any one or more of the methodologies or functions described herein. The instructions <NUM> (e.g., software) may also reside, completely or at least partially, within the main memory <NUM> or within the processor <NUM> (e.g., within a processor's cache memory) during execution thereof by the computer system <NUM>, the main memory <NUM> and the processor <NUM> also constituting machine-readable media. The instructions <NUM> (e.g., software) may be transmitted or received over a network <NUM> via the network interface device <NUM>.

Claim 1:
A method for establishing a virtual safety bubble (<NUM>) surrounding an autonomous farming machine (<NUM>) comprising a detection system (<NUM>) having a field of view of an environment surrounding the farming machine (<NUM>), wherein there are one or more blind spots (<NUM>) of the environment that are obstructed from view of the detection system (<NUM>), and the method comprising:
Prompting a manager of the farming machine (<NUM>) to confirm that there are no obstacles in the blind spots (<NUM>) of the detection system (<NUM>);
receiving a notification from the manager that there are no obstacles in the blind spots (<NUM>) of the detection system (<NUM>);
verifying that there are no obstacles (<NUM>, <NUM>) in unobstructed views of the detection system (<NUM>) by applying an obstacle detection model to images captured by the detection system;
determining a configuration of the farming machine (<NUM>) to autonomously perform farming actions in the environment with implements of the farming machine (<NUM>);
determining a virtual safety bubble (<NUM>) for the farming machine (<NUM> to autonomously perform the farming actions based on the determined configuration;
detecting an obstacle (<NUM>, <NUM>) in the environment by applying the obstacle detection model to images captured by the detection system (<NUM>);
determining that the obstacle (<NUM>, <NUM>) is within the virtual safety bubble (<NUM>); and
in response to determining that the obstacle (<NUM>, <NUM>) is within the virtual safety bubble (<NUM>), terminating autonomous operation of the farming machine (<NUM>).