DETECTING UNTRAVERSABLE ENVIRONMENT AND PREVENTING DAMAGE BY A VEHICLE

A vehicle moves through an environment (e.g., a farming, construction, mining, or forestry environment) and performs one or more actions in the environment. Portions of the environment may include moisture, such as puddles or mud patches. A control system associated with the vehicle may include a traversability model or a moisture model to help the vehicle operate in the environment with the moisture. In particular, the control system may employ the traversability model to reduce the likelihood of the vehicle attempting to traverse an untraversable portion of the environment, and the control system may employ the moisture model to reduce the likelihood of the vehicle performing an action that will damage a portion of the environment.

BACKGROUND

Field of Disclosure

This disclosure relates to operating a vehicle in an environment with moisture, and, more specifically, to preventing the vehicle from attempting to traverse untraversable areas in the environment or from damaging the environment.

Description of the Related Art

Operating a vehicle in an environment with moisture, such as puddles and mud, can pose difficulties for an operator of the vehicle. An environment with moisture can increase the likelihood of the vehicle becoming immobilized (e.g., getting stuck) in the environment or damaging the environment (e.g., forming a water run-off channel). An immobilized vehicle can be difficult to free, can delay operations, and can damage the environment, which may delay a timeline for a project for the environment. Preventing the vehicle from becoming immobilized or damaging the environment often requires knowledge of the capabilities of the vehicle and of the amount of moisture in the environment. This knowledge may be difficult to ascertain or may require the operator to have extensive working experience with the vehicle and the environment.

SUMMARY

A vehicle (e.g., a farming, construction, mining, or forestry vehicle) moves through an environment (e.g., a farming, construction, mining, or forestry environment) and performs one or more actions (e.g., farming, construction, mining, or forestry actions) in the environment. Portions of the environment may include moisture, such as puddles or mud patches. A control system associated with the vehicle may include a traversability model or a moisture model to help the vehicle operate in the environment.

In some embodiments, to reduce the likelihood of a vehicle becoming immobilized in an environment portion (e.g., due to moisture in the environment portion), the control system applies the traversability model to an image of the environment portion (the image may be captured by an image sensor of the vehicle). By analyzing pixels in the image, the traversability model determines a moisture level of the environment portion and determines a traversability difficulty of the environment portion using the moisture level. The traversability difficulty quantifies a level of difficulty for a vehicle to move through the portion of the environment. If the traversability difficulty is above a traversability capability of the vehicle, the vehicle performs an action, such as modifying the vehicle's route, so that it does not move through the portion of the environment.

In some embodiments, to reduce the likelihood of a vehicle damaging a portion of an environment (e.g., due to the moisture in the environment portion), the control system applies the moisture model to the image of the environment portion. The moisture model determines a measure of moisture for the environment portion of the environment using the image. Based on the determined measure of moisture, the control system determines a likelihood that the vehicle performing the action will damage the portion of the environment. If the likelihood is above a threshold likelihood, the vehicle performs another action, where the likelihood that the vehicle performing the other action will damage the portion of the environment is below the threshold likelihood.

The descriptions above are applicable to a variety of different environments and vehicles, such as construction vehicles (e.g., motor graders), agricultural or farming vehicles (e.g., tractors), or forestry vehicles (e.g., forwarders).

DETAILED DESCRIPTION

A vehicle (e.g., a farming, construction, mining, or forestry vehicle) includes one or more sensors capturing information about the surroundings as the vehicle moves through an environment. The environment can include various objects (e.g., ground and obstructions) used to determine actions (e.g., performing a treatment action, modifying a treatment parameter, modifying an operational parameter, and modifying a sensor parameter, etc.) for the vehicle to operate in the environment.

The vehicle includes a control system that processes the information obtained by the sensors to generate corresponding actions. For example, the control system processes information to identify objects to generate corresponding treatment actions. There are many examples of a vehicle (e.g., a farming vehicle) processing visual information obtained by an image sensor coupled to the vehicle to identify and treat plants and identify and avoid obstructions. For example, the vehicle as described in U.S. patent application Ser. No. 16/126,842 titled “Semantic Segmentation to Identify and Treat Plants in a Construction environment and Verify the Plant Treatments,” filed on Sep. 10, 2018, which is hereby incorporated by reference in its entirety.

II. Environment Management and Treatment Plans

Environment Management

Managers (e.g., agricultural, construction, mining, or forestry managers) are responsible for managing operations in one or more environments. Managers work to implement an objective (e.g., a farming, construction, mining, or forestry objective) within those environments and select from among a variety of actions (e.g., farming, construction, mining, or forestry actions) to implement that objective. Traditionally, managers are, for example, a human (e.g., agronomist) that works the environment (e.g., agricultural field) but could also be other systems configured to manage operations within the environment. For example, a manager could be an automated machine (e.g., vehicle), a machine learned computer model, etc. In some cases, a manager may be a combination of the managers described above. For example, a manager may include a human assisted by a machine learned model and one or more automated machines.

Managers implement one or more objectives for an environment. An objective is typically a macro-level goal for an environment. For example, macro-level farming objectives may include treating crops with growth promotors, neutralizing weeds with growth regulators, harvesting a crop with the best possible crop yield, or any other suitable farming objective. However, objectives may also be a micro-level goal for the environment. For example, micro-level farming objectives may include treating a particular plant in the environment, repairing, or correcting a part of a farming vehicle, requesting feedback from a manager, etc. Of course, there are many possible objectives and combinations of objectives, and the previously described examples are not intended to be limiting.

Objectives are accomplished (at least in part) by one or more vehicles performing a series of actions. Example, vehicles are described in greater detail below. Actions (e.g., farming, construction, mining, or forestry actions) are any operation implementable by a vehicle within the environment that works towards an objective. Consider, for example, a farming objective of harvesting a crop with the best possible yield. This farming objective requires a litany of farming actions, e.g., planting the environment, fertilizing the plants, watering the plants, weeding the environment, harvesting the plants, evaluating yield, etc. Similarly, each farming action pertaining to harvesting the crop may be a farming objective in and of itself. For instance, planting the environment (e.g., field) can require its own set of farming actions, e.g., preparing the ground (e.g., soil), digging in the ground, planting a seed, etc.

In other words, managers implement a treatment plan (e.g., farming, construction, mining, or forestry treatment plan) in the environment to accomplish an objective. A treatment plan is a hierarchical set of macro-level or micro-level objectives that accomplish the objective of the manager. Within a treatment plan, each macro or micro-objective may require a set of actions to accomplish, or each macro or micro-objective may be an action itself. So, to expand, the treatment plan is a temporally sequenced set of actions to apply to the environment that the manager expects will accomplish the objective.

When executing a treatment plan in an environment, the treatment plan itself or its constituent objectives and actions have various results. A result is a representation as to whether, or how well, a vehicle accomplished the treatment plan, objective, or action. A result may be a qualitative measure such as “accomplished” or “not accomplished,” or may be a quantitative measure such as “40 pounds harvested,” or “1.25 acres treated.” Results can also be positive or negative, depending on the configuration of the vehicle or the implementation of the treatment plan. Moreover, results can be measured by sensors of the vehicle, input by managers, or accessed from a datastore or a network.

Traditionally, managers have leveraged their experience, expertise, and technical knowledge when implementing actions in a treatment plan. In a first example, a manager may spot check weed pressure in several areas of the environment (e.g., field) to determine when an environment is ready for weeding. In a second example, a manager may refer to previous implementations of a treatment plan to determine the best time to begin planting an environment (e.g., field). In a third example, a manager may rely on established best practices in determining a specific set of farming actions to perform in a treatment plan to accomplish a farming objective.

Leveraging manager and historical knowledge to make decisions for a treatment plan affects both spatial and temporal characteristics of a treatment plan. For example, farming actions in a treatment plan have historically been applied to an entire environment (e.g., field) rather than small portions of the environment. To illustrate this example further, when a manager decides to plant a crop, they plant the entire environment instead of just a corner of the environment having the best planting conditions; or, when the manager decides to weed the environment, they weed the entire environment rather than just a few rows. Similarly, each action in a sequence of actions of a treatment plan are historically performed at approximately the same time. For example, when a manager decides to fertilize an environment (e.g., field), they fertilize the environment at approximately the same time; or, when the manager decides to harvest the environment, they do so at approximately the same time.

Notably though, vehicles have greatly advanced in their capabilities. For example, vehicles continue to become more autonomous, include an increasing number of sensors and measurement devices, employ higher amounts of processing power and connectivity, and implement various machine vision algorithms to enable managers to successfully implement a treatment plan.

Because of this increase in capability, managers are no longer limited to spatially and temporally monolithic implementations of actions in a treatment plan. Instead, managers may leverage advanced capabilities of vehicles to implement treatment plans that are highly localized and determined by real-time measurements in the environment. In other words, rather than a manager applying a “best guess” treatment plan to an entire environment, they can implement individualized and informed treatment plans for each plant in the environment.

III. Example Vehicles

FIG.1Ais a block diagram of a vehicle100(also referred to as a work vehicle) that performs actions of a treatment plan, according to an example embodiment. The vehicle100may be a vehicle used for farming (e.g., a tractor), construction (e.g., a motor grader), mining (e.g., a dragline excavator), or forestry (e.g., a forwarder). In the example ofFIG.1A, the vehicle100includes a detection mechanism110, a treatment mechanism120, a control system130, a mounting mechanism140, a coupling mechanism142, and a verification mechanism150. The described components and functions of the vehicle100are just examples, and a vehicle can have different or additional components and functions other than those described below. For example, the vehicle may also include a power source, digital memory, communication apparatus, or any other suitable component that enables the vehicle100to implement actions in a treatment plan.

Operating Environment

The vehicle100operates in an operating environment102(also referred to as the environment102). The environment102is a geographic area where the vehicle100implements actions of a treatment plan. Example environments include a farming field (indoor or outdoor), a construction site, a mining area, and a forest. An environment may include any number of environment portions. An environment portion is a subunit of an environment. The vehicle100can execute different actions for different environment portions. Moreover, an environment and an environment portion are largely interchangeable in the context of the methods and systems described herein. That is, treatment plans and their corresponding actions may be applied to an entire environment or an environment portion depending on the circumstances at play.

The operating environment102may include the ground and objects in, on, or above the ground. As such, actions the vehicle100implements as part of a treatment plan may be applied to the ground. The ground may include soil but can alternatively include sponge or any other suitable ground type.

The vehicle100may include a detection mechanism110. The detection mechanism110identifies objects in the operating environment102of the vehicle100. To do so, the detection mechanism110obtains information describing the environment102(e.g., sensor or image data), and processes that information to identify pertinent objects (e.g., plants, the ground, persons, etc.) in the operating environment102. Identifying objects in the environment102further enables the vehicle100to implement actions in the environment.

The vehicle100can include any number or type of detection mechanism110that may aid in determining and implementing actions. In some embodiments, the detection mechanism110includes one or more sensors. For example, the detection mechanism110can include a multispectral camera, a stereo camera, a CCD camera, a single lens camera, a CMOS camera, hyperspectral imaging system, LIDAR system (light detection and ranging system), a depth sensing system, dynamometer, IR camera, thermal camera, humidity sensor, light sensor, temperature sensor, or any other suitable sensor. Further, the detection mechanism110may include an array of sensors (e.g., an array of cameras) configured to capture information about the environment102surrounding the vehicle100. For example, the detection mechanism110may include an array of cameras configured to capture an array of pictures representing the environment102surrounding the vehicle100. The detection mechanism110may also be a sensor that measures a state of the vehicle100. For example, the detection mechanism110may be a speed sensor, a heat sensor, or some other sensor that can monitor the state of a component of the vehicle100.

A detection mechanism110may be mounted at any point on the mounting mechanism140. Depending on where the detection mechanism110is mounted relative to the treatment mechanism120, one or the other may pass over a geographic area in the environment before the other. For example, the detection mechanism110may be positioned on the mounting mechanism140such that it traverses over a geographic location before the treatment mechanism120as the vehicle100moves through the environment. In another examples, the detection mechanism110is positioned to the mounting mechanism140such that the two traverse over a geographic location at substantially the same time as the vehicle100moves through the environment. Similarly, the detection mechanism110may be positioned on the mounting mechanism140such that the treatment mechanism120traverses over a geographic location before the detection mechanism110as the vehicle100moves through the environment. The detection mechanism110may be statically mounted to the mounting mechanism140or may be removably or dynamically coupled to the mounting mechanism140. In other examples, the detection mechanism110may be mounted to some other surface of the vehicle100or may be incorporated into another component of the vehicle100.

The vehicle100may include a verification mechanism150. Generally, the verification mechanism150records a measurement of the operating environment102and the vehicle100may use the recorded measurement to verify or determine the extent of an implemented action (i.e., a result of the action).

To illustrate, consider an example where a vehicle100implements an action based on a measurement of the operating environment102by the detection mechanism110. The verification mechanism150records a measurement of the same geographic area measured by the detection mechanism110and where vehicle100implemented the determined action. The vehicle100then processes the recorded measurement to determine the result of the action. For example, the verification mechanism150may record an image of an object (e.g., tree) in a geographic region identified by the detection mechanism110and treated by a treatment mechanism120. The vehicle100may apply a treatment detection algorithm to the recorded image to determine the result of the treatment applied to the object.

Information recorded by the verification mechanism150can also be used to empirically determine operation parameters of the vehicle100that will obtain the desired effects of implemented actions (e.g., to calibrate the vehicle100, to modify treatment plans, etc.). For instance, the vehicle100may apply a calibration detection algorithm to a measurement recorded by the vehicle100. In this case, the vehicle100determines whether the actual effects of an implemented action are the same as its intended effects. If the effects of the implemented action are different than its intended effects, the vehicle100may perform a calibration process. The calibration process changes operation parameters of the vehicle100such that effects of future implemented actions are the same as their intended effects. To illustrate, consider the previous example where the vehicle100recorded an image of a treated object (e.g., a tree). There, the vehicle100may apply a calibration algorithm to the recorded image to determine whether the treatment is appropriately calibrated (e.g., at its intended location in the operating environment102). If the vehicle100determines that the vehicle100is not calibrated (e.g., the applied treatment is at an incorrect location), the vehicle100may calibrate itself such that future treatments are in the correct location. Other example calibrations are also possible.

The verification mechanism150can have various configurations. For example, the verification mechanism150can be substantially similar (e.g., be the same type of mechanism as) the detection mechanism110or can be different from the detection mechanism110. In some cases, the detection mechanism110and the verification mechanism150may be one in the same (e.g., the same sensor). In an example configuration, the verification mechanism150is positioned distal the detection mechanism110relative the direction of travel115, and the treatment mechanism120is positioned there between. In this configuration, the verification mechanism150traverses over a geographic location in the operating environment102after the treatment mechanism120and the detection mechanism110. However, the mounting mechanism140can retain the relative positions of the system components in any other suitable configuration. In some configurations, the verification mechanism150can be included in other components of the vehicle100.

The vehicle100can include any number or type of verification mechanism150. In some embodiments, the verification mechanism150includes one or more sensors. For example, the verification mechanism150can include a multispectral camera, a stereo camera, a CCD camera, a single lens camera, a CMOS camera, hyperspectral imaging system, LIDAR system (light detection and ranging system), a depth sensing system, dynamometer, IR camera, thermal camera, humidity sensor, light sensor, temperature sensor, or any other suitable sensor. Further, the verification mechanism150may include an array of sensors (e.g., an array of cameras) configured to capture information about the environment102surrounding the vehicle100. For example, the verification mechanism150may include an array of cameras configured to capture an array of pictures representing the operating environment102.

The vehicle100may include a treatment mechanism120. The treatment mechanism120can implement actions in the operating environment102of a vehicle100(although not all actions need to be performed by the treatment mechanism120). For instance, a vehicle100may include a treatment mechanism120that applies a treatment to an object in the operating environment102. More generally, the vehicle100employs the treatment mechanism120to apply a treatment to a treatment area, and the treatment area may include anything within the operating environment102(e.g., a plant or the ground). In other words, the treatment area may be any portion of the operating environment102.

If a treatment is a plant treatment, the treatment mechanism120applies a treatment to a plant in the environment. The treatment mechanism120may apply treatments to identified plants or non-identified plants. For example, the vehicle100may identify and treat a specific plant in the environment. Alternatively, or additionally, the vehicle100may identify some other trigger that indicates a plant treatment and the treatment mechanism120may apply a plant treatment. Some example plant treatment mechanisms120include: one or more spray nozzles, one or more electromagnetic energy sources (e.g., a laser), one or more physical implements configured to manipulate plants, but other plant treatment mechanisms120are also possible.

If the treatment is a ground treatment, the treatment mechanism120applies a treatment to some portion of the ground in the environment. The treatment mechanism120may apply treatments to identified areas of the ground, or non-identified areas of the ground. For example, the vehicle100may identify and treat an area of ground in the environment. Alternatively, or additionally, the vehicle100may identify some other trigger that indicates a ground treatment and the treatment mechanism120may apply a treatment to the ground. Some example treatment mechanisms120configured for applying treatments to the ground include: one or more spray nozzles, one or more electromagnetic energy sources, one or more physical implements configured to manipulate the ground (e.g., an excavator tool or pile driver tool), but other ground treatment mechanisms120are also possible.

Of course, the vehicle100is not limited to treatment mechanisms120for plants and the ground. The vehicle100may include treatment mechanisms120for applying various other treatments to objects in the environment.

Depending on the configuration, the vehicle100may include various numbers of treatment mechanisms120(e.g., 1, 2, 5, 20, 60, etc.). A treatment mechanism120may be fixed (e.g., statically coupled) to the mounting mechanism140or attached to the vehicle100. Alternatively, or additionally, a treatment mechanism120may be movable (e.g., translatable, rotatable, etc.) on the vehicle100. In one configuration, the vehicle100includes a single treatment mechanism120. In this case, the treatment mechanism120may be actuatable to align the treatment mechanism120to a treatment area122. In a second variation, the vehicle100includes a treatment mechanism120assembly comprising an array of treatment mechanisms120. In this configuration, a treatment mechanism120may be a single treatment mechanism120, a combination of treatment mechanisms120, or the treatment mechanism120assembly. Thus, either a single treatment mechanism120, a combination of treatment mechanisms120, or the entire assembly may be selected to apply a treatment to a treatment area. Similarly, either the single, combination, or entire assembly may be actuated to align with a treatment area, as needed. In some configurations, the vehicle100may align a treatment mechanism120with an identified object in the operating environment102. That is, the vehicle100may identify an object in the operating environment102and actuate the treatment mechanism120such that its treatment area aligns with the identified object.

A treatment mechanism120may be operable between a standby mode and a treatment mode. In the standby mode the treatment mechanism120does not apply a treatment, and in the treatment mode the treatment mechanism120is controlled by the control system130to apply the treatment. However, the treatment mechanism120can be operable in any other suitable number of operation modes.

The vehicle100includes a control system130. The control system130controls operation of the various components and systems on the vehicle100. For instance, the control system130can obtain information about the operating environment102, processes that information to identify an action to implement, and implement the identified action with system components of the vehicle100.

The control system130can receive information from the detection mechanism110, the verification mechanism150, the treatment mechanism120, or any other component or system of the vehicle100. For example, the control system130may receive measurements from the detection mechanism110or verification mechanism150, or information relating to the state of a treatment mechanism120or implemented actions from a verification mechanism150. Other information is also possible.

Similarly, the control system130can provide input to the detection mechanism110, the verification mechanism150, or the treatment mechanism120. For instance, the control system130may be configured to input and control operating parameters of the vehicle100(e.g., speed or direction). Similarly, the control system130may be configured to input and control operating parameters of the detection mechanism110or verification mechanism150. Operating parameters of the detection mechanism110or verification mechanism150may include processing time, location, or angle of the detection mechanism110, image capture intervals, image capture settings, etc. Other inputs are also possible. The control system may be configured to generate machine inputs for the treatment mechanism120. That is translating an action of a treatment plan into machine instructions implementable by the treatment mechanism120.

The control system130can be operated by a user operating the vehicle100, wholly or partially autonomously, operated by a user connected to the vehicle100by a network, or any combination of the above. For instance, the control system130may be operated by a manager sitting in a cabin of the vehicle100, or the control system130may be operated by a manager connected to the control system130via a wireless network. In another example, the control system130may implement an array of control algorithms, machine vision algorithms, decision algorithms, etc. that allow it to operate autonomously or partially autonomously.

The control system130may be implemented by a computer or a system of distributed computers. The computers may be connected in various network environments. For example, the control system130may be a series of computers implemented on the vehicle100and connected by a local area network. In another example, the control system130may be a series of computers implemented on the vehicle100, in the cloud, a client device and connected by a wireless area network.

The control system130can apply one or more computer models to determine and implement actions in the environment. For example, in an example farming context, the control system130can apply a plant identification module to images acquired by the detection mechanism110to determine and implement actions. The control system130may be coupled to the vehicle100such that an operator (e.g., a driver) can interact with the control system130. In other embodiments, the control system130is physically removed from the vehicle100and communicates with system components (e.g., detection mechanism110, treatment mechanism120, etc.) wirelessly.

In some configurations, the vehicle100may additionally include a communication apparatus, which functions to communicate (e.g., send or receive) data between the control system130and a set of remote devices. The communication apparatus can be a Wi-Fi communication system, a cellular communication system, a short-range communication system (e.g., Bluetooth, NFC, etc.), or any other suitable communication system.

Other Vehicle Components

In various configurations, the vehicle100may include any number of additional components.

For instance, the vehicle100may include a mounting mechanism140. The mounting mechanism140provides a mounting point for the components of the vehicle100. That is, the mounting mechanism140may be a chassis or frame to which components of the vehicle100may be attached but could alternatively be any other suitable mounting mechanism140. More generally, the mounting mechanism140statically retains and mechanically supports the positions of the detection mechanism110, the treatment mechanism120, and the verification mechanism150.

The vehicle100may include locomoting mechanisms. The locomoting mechanisms may include any number of wheels, continuous treads, articulating legs, or some other locomoting mechanism(s). For instance, the vehicle100may include a first set and a second set of coaxial wheels, or a first set and a second set of continuous treads. In the either example, the rotational axis of the first and second set of wheels/treads are approximately parallel. Further, each set may be arranged along opposing sides of the vehicle100. Typically, the locomoting mechanisms are attached to a drive mechanism that causes the locomoting mechanisms to translate the vehicle100through the operating environment102. For instance, the vehicle100may include a drive train for rotating wheels or treads. In different configurations, the vehicle100may include any other suitable number or combination of locomoting mechanisms and drive mechanisms.

The vehicle100may also include one or more coupling mechanisms142(e.g., a hitch). The coupling mechanism142functions to removably or statically couple various components of the vehicle100. For example, a coupling mechanism may attach a drive mechanism to a secondary component such that the secondary component is pulled behind the vehicle100. In another example, a coupling mechanism may couple one or more treatment mechanisms120to the vehicle100.

The vehicle100may additionally include a power source, which functions to power the system components, including the detection mechanism110, control system130, and treatment mechanism120. The power source can be mounted to the mounting mechanism140, can be removably coupled to the mounting mechanism140, or can be incorporated into another system component (e.g., located on the drive mechanism). The power source can be a rechargeable power source (e.g., a set of rechargeable batteries), an energy harvesting power source (e.g., a solar system), a fuel consuming power source (e.g., a set of fuel cells or an internal combustion system), or any other suitable power source. In other configurations, the power source can be incorporated into any other component of the vehicle100.

Example vehicles100configured for various environments are further described below with reference toFIGS.1B-1G.

III.A Example Farming Vehicles

An example embodiment of vehicle100is a farming vehicle. A farming vehicle is a vehicle configured to operate in a farming environment and to accomplish (or contribute to accomplishing) one or more objectives in the farming environment. A farming action may be any operation implementable by a farming vehicle within the farming environment that works towards the one or more objectives. Farming vehicles can include a wide variety of vehicles (e.g., tractors, drapers, balers, tillers, and harvesters) which can perform a variety of farming actions (e.g., planting, spraying, weeding, pruning, and harvesting) in farming treatment plans to accomplish farming objectives (e.g., planting a field or applying a pesticide to a field). An example farming environment is a field (e.g., for growing crops).

FIGS.1B-1Dillustrate example farming vehicles (100A,100B), according to some embodiments. Specifically,FIG.1Bis an isometric view of a tractor farming vehicle100A that performs farming actions of a treatment plan, according to one example embodiment, andFIG.1Cis a top view of the farming vehicle100A inFIG.1B.FIG.1Dis an isometric view of another tractor farming vehicle100B that performs farming actions of a treatment plan, in accordance with one example embodiment. As illustrated, the farming vehicles (100A,100B) each include a detection mechanism (110A,110B), a treatment mechanism (120A,120B), a control system (130A,130B), a mounting mechanism (140A,140B), a coupling mechanism (142A,142B), and a verification mechanism (150A,150B), which are example embodiments of the corresponding components inFIG.1A.

The farming vehicles inFIGS.1B-1Dare each configured to implement a farming action which applies a treatment to one or more plants104or the ground106within the environment102. A treatment farming action may be included in a treatment plan to regulate plant growth. As such, treatments may be applied directly to a single plant104but can alternatively be directly applied to multiple plants104, indirectly applied to one or more plants104, applied to the environment102associated with the plant104(e.g., soil, atmosphere, or other suitable portion of the plant's environment adjacent to or connected by an environmental factors, such as wind), or otherwise applied to the plants104.

If the treatment is a plant treatment, the effect of treating a plant with a treatment mechanism (e.g.,120A) may include any of plant necrosis, plant growth stimulation, plant portion necrosis or removal, plant portion growth stimulation, or any other suitable treatment effect. Moreover, the treatment mechanism can apply a treatment that dislodges a plant104from the ground106, severs a plant104or portion of a plant104(e.g., cutting), incinerates a plant104or portion of a plant104, electrically stimulates a plant104or portion of a plant104, fertilizes or promotes growth (e.g., with a growth hormone) of a plant104, waters a plant104, applies light or some other radiation to a plant104, or injects one or more working fluids into the ground106adjacent to a plant104(e.g., within a threshold distance from the plant). Other plant treatments are also possible. When applying a plant treatment, the treatment mechanisms may be configured to spray one or more of: an herbicide, a fungicide, insecticide, some other pesticide, or water.

In a particular example, the farming vehicle is configured to implement an action which applies a treatment that necroses the entire plant104(e.g., weeding) or part of the plant104(e.g., pruning). In this case, the action can include dislodging the plant104from the ground106, incinerating a portion of the plant104(e.g., with directed electromagnetic energy such as a laser), applying a treatment concentration of working fluid (e.g., fertilizer, hormone, water, etc.) to the plant104, or treating the plant104in any other suitable manner. In another example, a farming vehicle (e.g.,100A) is configured to implement an action which applies a treatment to regulate plant growth. Regulating plant growth can include promoting plant growth, promoting growth of a plant portion, hindering (e.g., retarding) plant104or plant portion growth, or otherwise controlling plant growth. Examples of regulating plant growth includes applying growth hormone to the plant104, applying fertilizer to the plant104or ground106, applying a disease treatment or insect treatment to the plant104, electrically stimulating the plant104, watering the plant104, pruning the plant104, or otherwise treating the plant104. Plant growth can additionally be regulated by pruning, necrosing, or otherwise treating the plants104adjacent to the plant104.

In the examples ofFIGS.1B-1D, the mounting mechanism (140A,140B) extends outward from a body of the farming vehicle (100A,100B) such that the mounting mechanism140is approximately perpendicular to the direction of travel115. In some configurations, the mounting mechanism (140A,140B) may include an array of treatment mechanisms (120A,120B) positioned laterally along the mounting mechanism (140A,140B). In some configurations, the farming vehicle (100A,100B) may not include a mounting mechanism (140A,140B), the mounting mechanism (140A,140B) may be alternatively positioned, or the mounting mechanism (140A,140B) may be incorporated into any other component of the vehicle (100A,100B).

The plants104can be crops but can also be weeds or any other suitable plant104. Some example crops include cotton, lettuce, soybeans, rice, carrots, tomatoes, corn, broccoli, cabbage, potatoes, wheat, or any other suitable commercial crop. The weeds may be grasses, broadleaf weeds, thistles, or any other suitable determinantal weed. More generally, plants104may include a stem that is arranged superior to (e.g., above) the ground106and a root system joined to the stem that is located inferior to the plane of the ground106(e.g., below ground). The stem may support any branches, leaves, or fruits. The plant104can have a single stem, leaf, or fruit, multiple stems, leaves, or fruits, or any number of stems, leaves or fruits. The root system may be a tap root system or fibrous root system, and the root system may support the plant104position and absorb nutrients and water from the ground106. In various examples, the plant104may be a vascular plant104, non-vascular plant104, ligneous plant104, herbaceous plant104, or be any suitable type of plant104.

Plants104in an environment may be grown in one or more plant104rows (e.g., plant104beds). The plant104rows are typically parallel to one another but do not have to be. Each plant104row is generally spaced between 2 inches and 45 inches apart when measured in a perpendicular direction from an axis representing the plant104row. Plant104rows can have wider or narrower spacings or could have variable spacing between multiple rows (e.g., a spacing of 12 in. between a first and a second row, a spacing of 16 in. a second and a third row, etc.).

Plants104within an environment may include the same type of crop (e.g., same genus, same species, etc.). For example, each portion in an environment may include corn crops. However, the plants104within each environment may also include multiple crops (e.g., a first, a second crop, etc.). For example, some environment portions may include lettuce crops while other environment portions include pig weeds, or, in another example, some environment portions may include beans while other environment portions include corn. Additionally, a single environment portion may include different types of crops. For example, a single environment portion may include a soybean plant104and a grass weed.

III.B Example Construction Vehicles

Another example embodiment of vehicle100is a construction vehicle. A construction vehicle is a vehicle configured to operate in a construction environment and to accomplish (or contribute to accomplishing) one or more objectives in the construction environment. A construction action may be any operation implementable by a construction vehicle within the construction environment that works towards the one or more objectives. Construction vehicles can include a wide variety of vehicles (e.g., bulldozers, front loaders, dump trucks, backhoes, graders, trenchers, cranes, loaders, crawler dozers, compactors, forklifts, conveyors, and mixer trucks) which can perform a variety of construction actions (e.g., excavating, pile driving, loading objects, unloading objects, lifting objects, clearing debris, grading, and digging trenches) in construction treatment plans to accomplish construction objectives (e.g., building a road, digging a trench, digging a hole, clearing a portion of dirt, or moving dirt from point A to point B).

An example construction environment that a construction vehicle can operate in is a construction site or project site. A construction environment may be an area used to construct, repair, maintain, improve, extend, or demolish buildings, infrastructure, or industrial facilities. A construction environment may include one or more of the following: a secure perimeter to restrict unauthorized access, site access control points, office and welfare accommodation for personnel from the main contractor and other firms involved in the project team, or storage areas for materials, machinery (e.g., construction vehicles), or equipment. In some cases, a construction environment is formed when the first feature of a permanent structure has been put in place, such as pile driving, or the pouring of slabs or footings.

FIGS.1E and1Fillustrate example construction vehicles (100C,100D), according to some embodiments. Specifically,FIG.1Eis an isometric view of a wheel loader construction vehicle100C, andFIG.1Fis an isometric view of a dump truck construction vehicle100D. As illustrated, each construction vehicle (100C,100D) each include a detection mechanism (110C,110D), a treatment mechanism (120C,120D), a control system (130C,130D), a mounting mechanism (140C,140D), a coupling mechanism (142C,142D), and a verification mechanism (150C,150D), which are example component embodiments of the corresponding components inFIG.1A.

In one example situation, the loader100C inFIG.1Eis engaged in moving material from a pile to the treatment mechanism120D (e.g., cargo bed) of the dump truck100D inFIG.1F. To fill the truck100D, the loader100C starts by moving forward along a path to pick up a load and, once at the pile, digs the treatment mechanism120C (e.g., bucket) into the pile to fill the treatment mechanism120C with material. Then, the loader100C backs away from the pile, while turning to face the dump truck100D. Then the loader drives to the dump truck100D, raising its treatment mechanism120C (e.g., bucket) and dumps the material into the treatment mechanism120D of the dump truck100D. Afterwards, the loader100C backs up and turns to face the pile, repeating the process. As further described below, the loader100C may encounter moisture during any of these construction actions or before or after the completion of the objective, in between objectives, or in other scenarios. The loader100C or the dump truck100D may employ a control system (130C,130D) to identify moisture in the environment. For instance, the control system130C may employ a traversability model to reduce the likelihood of the loader100C becoming immobilized (e.g., getting stuck) in terrain, and may employ a moisture model to reduce the likelihood of the loader100C performing an action that will damage the environment.

III. C Example Forestry Vehicles

Another example embodiment of vehicle100is a forestry vehicle. A forestry vehicle is a vehicle configured to operate in a forestry environment and to accomplish (or contribute to accomplishing) one or more objectives in the forestry environment. A forestry action may be any operation implementable by a forestry vehicle within the forestry environment that works towards the one or more objectives. Forestry vehicles can include a wide variety of vehicles (e.g., harvesters, skidders, feller bunchers, forwarders, and swing machines) which can perform a variety of forestry actions (e.g., felling, delimbing, bucking, forwarding, and sorting) in forestry treatment plans to accomplish forestry objectives (e.g., trimming a set of trees, felling a set of trees, or moving logs from point A to point B).

Example forestry environments that forestry vehicles can operates in include a forest, a woodland, and a logging site. A forestry environment may be an area used to create, manage, use, plant, conserve, or repair forests or woodlands. Generally, a forest is an area of land dominated by trees. For example, a forest is an area of land spanning more than 0.5 hectares with trees higher than 5 meters and a canopy cover of more than 10 percent, or trees able to reach these thresholds in situ. Typically, a forest does not include land that is predominantly under agricultural or urban use.

FIG.1Gillustrates an example forestry vehicle100E, according to an embodiment. Specifically,FIG.1Gis a side view of a skidder forestry vehicle100E. As illustrated, the forestry vehicle100E includes a detection mechanism110E, a treatment mechanism120E, a control system130E, a mounting mechanism140E, a coupling mechanism142E, and a verification mechanism150E, which are example component embodiments of the corresponding components inFIG.1A.

IV. System Environment

FIG.2is a block diagram of the system environment for the vehicle100, in accordance with one or more example embodiments. In this example, the control system210(e.g., control system130) is connected to external systems220and a vehicle component array230via a network240within the system environment200.

The external systems220are any system that can generate data representing information useful for determining and implementing actions in an environment. External systems220may include one or more sensors222, one or more processing units224, and one or more datastores226. The one or more sensors222can measure the environment102, the vehicle100, etc. and generate data representing those measurements. For instance, the sensors222may include a rainfall sensor, a wind sensor, heat sensor, a camera, etc. The processing units2240may process measured data to provide additional information that may aid in determining and implementing actions in the environment. For instance, a processing unit224may access an image of an environment and may access historical weather information for an environment to generate a forecast for the environment. Datastores226store historical information regarding the vehicle100, the operating environment102, etc. that may be beneficial in determining and implementing actions. For instance, the datastore226may store results of previously implemented treatment plans and actions for an environment, a nearby environment, or the region. The historical information may have been obtained from one or more vehicles (i.e., measuring the result of an action from a first vehicle with the sensors of a second vehicle). Further, the datastore226may store results of specific actions in the environment, or results of actions taken in nearby environments having similar characteristics. The datastore226may also store historical weather, flooding, environment use, objects in the environment, etc. for the environment and the surrounding area. Finally, the datastores226may store any information measured by other components in the system environment200.

The vehicle component array230includes one or more components232. Components232are elements of the vehicle100that can take actions (e.g., a treatment mechanism120). As illustrated, each component has one or more input controllers234and one or more sensors236, but a component may include only sensors236or only input controllers234. An input controller234controls the function of the component232. For example, an input controller234may receive machine commands via the network240and actuate the component232in response. A sensor236generates data representing measurements of the operating environment and provides that data to other systems and components within the system environment200. The measurements may be of a component232, the vehicle100, the operating environment, etc. For example, a sensor236may measure a configuration or state of the component232(e.g., a setting, parameter, power load, etc.), measure conditions in the operating environment (e.g., moisture, temperature, etc.), capture information representing the operating environment (e.g., images, depth information, distance information), and generate data representing the measurement(s).

The control system210receives information from external systems220and the vehicle component array230and implements a treatment plan in an environment with a vehicle. The control system210may include one or more models and instructions to operate the vehicle in an environment with moisture. For example, inFIG.2, the control system210includes a traversability model240and a moisture model250. The traversability model240and moisture model250are further described below.

The network240connects nodes of the system environment200to allow microcontrollers and devices to communicate with each other. In some embodiments, the components are connected within the network as a Controller Area Network (CAN). In this case, within the network each element has an input and output connection, and the network240can translate information between the various elements. For example, the network240receives input information from the external system220, processes the information, and transmits the information to the control system210. The control system210generates an action based on the information and transmits instructions to implement the action to the appropriate component(s)232of the component array230.

Additionally, the system environment200may be other types of network environments and include other networks, or a combination of network environments with several networks. For example, the system environment200, can be a network such as the Internet, a LAN, a MAN, a WAN, a mobile wired or wireless network, a private network, a virtual private network, a direct communication line, and the like.

V. Operating a Vehicle in an Environment with Moisture

As described above, a vehicle (e.g., vehicle100) is configured to move through an environment and perform one or more actions (e.g., farming, construction, forestry, or mining actions) in the environment. Portions of the environment may include moisture, such as puddles or mud patches. A control system (e.g., control system130) associated with the vehicle may include one or more models to help the vehicle operate (e.g., perform one or more actions) in the environment with moisture. In particular, the control system may employ a traversability model to reduce the likelihood of the vehicle becoming immobilized (e.g., getting stuck) in a portion of the environment, and may employ a moisture model to reduce the likelihood of the vehicle performing an action that will damage a portion of the environment. For example, in a construction context, a pit (e.g., at a rock quarry) may fill with water due to rain. The control system may use the traversability model to determine whether a heavy dump truck can drive through the pit without getting stuck and may use the moisture model to determine if the heavy dump truck will further enlarge the pit by an undesirable amount by driving through the pit. In another example, in another construction context, a mound of dirt may get wet due to rain. The control system may use the traversability model to determine whether a construction vehicle can drive up the mound without getting stuck, slipping back down the mound, or losing traction. Additionally, or alternatively, the control system may use the moisture model to determine if the mound will get damaged (e.g., deformed) due to a construction vehicle driving up the mound.

FIG.3is an overhead view of a farming vehicle300(an example of a vehicle100) moving along a route310through an agricultural field340with moisture (an example of an environment with moisture), according to an embodiment. Portions of the field (e.g., portions325A and325B) include moisture in the form of puddles of liquid320and a mud patch327. In the example ofFIG.3, puddle320B is significantly larger than puddle320A. The farming vehicle300(e.g., via the control system130) can analyze the moisture in the field340and determine whether the farming vehicle300will get stuck or damage the field340as it moves through portions325of the field340that include the puddles320and mud patch327. In the example ofFIG.3, the control system determines that the vehicle can move through the portion325A (including puddle320A and mud patch327). However, the farming vehicle300determines that it cannot or should not move through the portion325B (including puddle320B). For example, the farming vehicle300determines that the portion325B has a traversability difficulty above the traversability capability of the farming vehicle300. In another example, the farming vehicle300determines that the likelihood of the farming vehicle300damaging the portion325B is above a likelihood threshold.

In response to the farming vehicle300determining that it cannot or should not move through the portion325B (e.g., the vehicle determines the portion325B is untraversable or there is a high likelihood that the vehicle will damage the portion325B), the vehicle generates a modified route330for the farming vehicle300. By traveling along the modified route330, the farming vehicle300will avoid the portion of the field340that includes puddle320B. If the farming vehicle300is performing actions in the field340(e.g., a farming action), the route may be modified so that portions of the field340around puddle320B still receive actions by the vehicle (e.g., portions are still treated with a pesticide). For example, the farming vehicle300may drive around the puddle and back up to the puddle320B to reduce the amount of unworked ground around the puddle320B.

AlthoughFIG.3and its above description are in a farming context, they are not limited to this context. The concepts described above with respect toFIG.3can be applied to other contexts where a vehicle is moving through an environment with moisture, such as a construction vehicle moving through a construction site with moisture or a forestry vehicle moving through a forest with moisture. For example, a grader construction vehicle applying grading construction actions on a construction site may operate in a similar manner as the farming machine300.

V.A Applying the Traversability Model

FIG.4illustrates a method for operating in an environment (e.g., field, road, street, construction site, mining site or forest) with moisture by a vehicle (e.g., vehicle100), in accordance with an example embodiment. One or more steps of method400may be performed from the perspective of the control system130. The method400can include greater or fewer steps than described herein. Additionally, the steps can be performed in different order, or by different components than described herein.

A vehicle (e.g., vehicle100) moves410along a route in an environment (e.g., a farming, construction, mining, or forestry environment) towards a portion of the environment including moisture. An example of this is illustrated inFIG.3. As described herein, a portion of the environment (also referred to as an environment portion) is a subsection of the environment that is smaller than the entire environment. An environment portion is large enough to include one or more bodies of moisture, which are large enough for the vehicle to potentially get stuck or large enough that the vehicle can potentially damage the environment if it moves through the environment portion. The vehicle may be actively controlled by an operator in the vehicle, remotely controlled by an operator, or autonomous. If the vehicle is autonomous, it may still receive instructions from an operator.

The control system accesses420an image of the portion of the environment. The image includes a group of pixels that indicate a moisture level of the portion of the environment. One or more image sensors capture the image. Example image sensors that can capture the image are described with reference to the detection mechanism110. The image sensors may be coupled to the vehicle and oriented to capture the portion of the environment (e.g., a portion of the environment in front of the vehicle). The image sensors may capture images as the vehicle moves along the route.FIGS.5A-5Eare example images of environments with moisture that may be accessed by the control system.FIGS.5A-5Eare further described with reference to step430.

Returning toFIG.4, the control system applies430a traversability model to the image of the portion of the environment. The traversability model determines a moisture level of the portion of the environment and determines a traversability difficulty for the portion of the environment using the moisture level. Additionally, or alternatively, in some embodiments, the traversability model determines whether an environment portion is traversable or untraversable. Determining whether an environment portion is traversable or untraversable may be based on the moisture level. If a traversable portion of the environment is detected, the vehicle may move through the detected portion. If an untraversable portion of the environment is detected, an obstacle event may be triggered so that the vehicle performs an action, such as modifying the vehicle's route so that it does not move through the detected untraversable portion.

Moisture as described herein can include liquid (e.g., water) on the surface of the ground (e.g., a puddle, body, or pool of water), liquid in the ground (e.g., mud), and liquid in the air (e.g., rain or fog). The moisture level (also referred to as a measure of moisture) describes an amount of moisture in, on, or above the ground (e.g., soil) in the environment portion. The level may be an objective measure, such as an estimate in gallons of the amount of moisture or the shape and size of a body of moisture (e.g., the depth, width, and length a body of liquid). The level can alternatively be on a scale, such as one to ten, where one indicates no moisture and ten indicates the presence of a large amount of moisture. If the portion of the environment includes multiple bodies of moisture, the traversability model may determine multiple moisture levels e.g., a moisture level for each body of moisture. In some embodiments, the traversability model distinguishes between liquid on the surface, liquid in the ground, and liquid in the air and determines a moisture level for each. For example, the control system determines a moisture level for a pool of liquid on the surface and determines another level for mud around the pool of liquid. In some embodiments, the traversability model detects any possible obstructions due to moisture and quantifies how much of it is on a path of the vehicle or the percentage of the obstacle in the FOV (field of view) of the image sensor.

The control system determines the moisture level for the environment portion in the image, for example, by applying a moisture model. The control system determines the moisture level by analyzing one or more groups of pixels in the image to identify moisture and determine an amount of moisture in the image. For example, visual properties such as texture, reflection, and saturation indicate the presence, location, and amount of moisture. In some embodiments, the detection of polarized light may be used to detect the presence of liquid. In another example, pixel values from a thermal sensor are analyzed (e.g., since moisture can be identified by comparing local temperature values). In some embodiments, multiple images are used by the traversability model. For example, images captured by different types of image sensors or images captured at different views are analyzed together to determine the moisture level.

In addition to analyzing pixels of the image, the traversability model may receive non-visual information to determine the moisture level of the portion of the environment, such as temperature, humidity, wind, weather data, topography, or soil maps. For example, the control system accesses current or historical weather data for the portion of the environment to determine the moisture level.

As stated earlier, the traversability model uses the moisture level to determine a traversability difficulty for the portion of the environment. The traversability difficulty quantifies a level of difficulty for a vehicle to move through the portion of the environment having the moisture level. As described herein, a higher traversability difficulty indicates an environment portion is less traversable and a lower traversability difficulty indicates an environment portion is more traversable. Generally, a high moisture level results in a high traversability difficulty and vice versa, however the relationship may not be linear, and the traversability difficulty may depend on other factors, some of which are further described below. The relationship between moisture level and traversability difficulty may be machine learned, for example, by training the traversability model with historical traversability data. Historical traversability data may include images of environment portions, moisture levels of moisture in the images, and traversability difficulty scores associated with the environment portions. In some embodiments, the traversability model is trained for the specific environment that the vehicle is moving through (e.g., agricultural fields, constructions sites, or forestry environments). For example, a large puddle in a quarry may be traversable (e.g., since the ground includes gravel or concrete), while a similarly sized puddle in a farming field may not be traversable (e.g., since the ground includes mud). In these embodiments, the historical traversability data may include images of the specific environment (e.g., images of construction sites with moisture). The traversability difficulty is generally determined prior to the vehicle moving through the environment portion. However, a traversability difficulty may be determined or updated if/when the vehicle moves through the environment portion.

In some embodiments, the traversability difficulty indicates a likelihood of a vehicle losing traction or getting stuck. In another example, the traversability difficulty is on a scale, such as one to ten, where one indicates almost any vehicle can move through the environment portion and ten indicates only highly specialized vehicles can move through the environment portion. In other embodiments, the traversability difficulty specifies characteristics of vehicles that can move through the portion of the environment. For example, the traversability difficulty specifies a wheel type (e.g., wheel or track), a wheel size, a tread type, an engine/motor type, a drive type (e.g., front, rear, or all-while drive), a make, a model, a weight, a treatment mechanism, or coupling mechanism of a vehicle that can move through the portion of the environment.

While the traversability difficulty is based on the moisture level of the environment portion, the traversability difficulty may also be based on additional factors, such as ground (e.g., soil) type or gradient. For example, the traversability model includes a weighted model with a weight for each factor, where each weight indicates how strongly its corresponding factor affects the traversability difficulty. The additional factors may be determined by the traversability model. Example additional factors are described below.

An example additional factor is the one or more ground (e.g., soil) types in the portion of the environment. One or more ground types may be determined by analyzing pixels of an image of the environment portion (ground types may have identifiable colors and textures), accessing a soil map, or receiving input from an operator of the vehicle. Example ground types include clay, loam, sand, silt, gravel, asphalt, and concrete. Since moisture (and the amount of moisture) may affect the traversability of ground types differently, determining a ground type of an environment portion can assist in determining the traversability difficulty. For example, moisture in sand generally has no effect on traversability, but moisture in clay or loam generally decreases traversability (i.e., increases the traversability difficulty). If multiple ground types are identified, the traversability of the combination of the ground types may be considered (e.g., the presence of gravel in clay may make it more traversable). A ground type may also include objects (e.g., debris) on the ground that may affect traversability. For example, large amounts of leaves and branches on an asphalt road may make the road more slippery.

Another example factor is the gradient of the environment portion of the environment (also referred to as the grade or slope). Generally, higher a gradient decreases the traversability for an environment portion. In some embodiments, a slope larger than 9 degrees renders the environment portion untraversable. The gradient may be determined by analyzing pixels of the image, accessing a topography map, or receiving input from an operator of the vehicle.

Other indicators of the traversability difficulty include:

(1) The visibility of an edge of a body of moisture (also referred to as the boundary or outline). If an edge of a body of moisture is visible and distinct, it may indicate that the ground around the body is firm and dry. Thus, an identifiable edge of a body may decrease the traversability difficulty.

(2) An amount of debris (e.g., plant matter or construction debris) in a body of moisture. The presence of debris may decrease the traversability difficulty because debris may reduce the likelihood of loss of traction. Additionally, the presence of debris sticking up through a body of liquid may indicate that the body is not deep.

(3) A depth of track marks. The depth of track marks may indicate how firm the ground (e.g., soil) is. Deep track marks may indicate an environment portion is less traversable, and shallow track marks may indicate an environment portion is more traversable. The size of dirt clods (e.g., made by the vehicle as it moves through the environment) may also indicate how firm the ground is. For example, larger dirt clods may indicate an environment portion includes more moisture and is less traversable and smaller dirt clods my indicate an environment portion includes less moisture and is more traversable.

(4) Movement of a body of liquid. Movement of liquid can make an environment portion more difficult to traverse. Thus, a stagnant or slow-moving body may have a lower traversability difficulty than a body with a current (e.g., a river or stream).

These factors may be determined by analyzing pixels in an image of the environment portion. In some embodiments, one or more of these factors are part of or contribute to the moisture level. In some embodiments, the traversability difficulty is also based on factors that are not related to moisture, such as the presence of obstacles in the environment portion (e.g., a boulder or trench). Descriptions of the moisture level, additional factors, and the traversability difficulty are further described below with reference toFIGS.5A-5E.

FIG.5Ais an image of an environment505that includes a mud patch510with water515in the lower right corner. In the example, ofFIG.5A, the environment505is a farming field with rows. The amount of moisture may be determined based on the area and depth of the mud patch510. The area may be determined by comparing the size of the patch510to the size of the rows in the environment. Similarly, the depth of the mud patch may be determined by comparing the height of the rows relative to the surface of the water515and mud in the patch510. The depth of the water515does not seem deep since water in the mud is generally below the tops of the rows. Since the edges of the mud patch510are not clearly identifiable, this may increase the traversability difficulty of the mud patch. Although the environment is a farming field, the considerations described above (e.g., how the amount of moisture is determined) are applicable in other contexts, such as construction, mining, and forestry.

FIG.5Bis an image of a dirt road that includes circular puddles520. The road may be part of many environment types, such as on a farming field, at a construction site, or in the forest. The edges of the puddles520are distinct, which may indicate that the ground around the puddles520is firm and dry. This is supported by the presence of faint track marks525in the dirt around the puddles520. Additionally, the size of the puddles520is small (e.g., determined by comparing the puddles520to the width of the road). All of these features indicate that the moisture level and traversability difficulty are low for the road inFIG.5B.

FIG.5Cis another image of a road530(e.g., at a farming field, at a construction site, or in the forest). The road530is muddy and includes puddles535with edges that are less defined than inFIG.5B. This indicates that the road530is less traversable than the road inFIG.5B. However, the mud and puddles include plant matter540, which increases the traversability.FIG.5Calso includes track marks545that are deeper than the track marks525inFIG.5B, which may decrease the traversability.

FIGS.5D and5Eare images of farming fields with large puddles (550and560). Generally, small puddles between farm field rows are traversable. However, if standing water goes above the rows (e.g., rows555and565), the environment may become non-traversable.

Referring back toFIG.4, the vehicle performs440an action (e.g., after receiving instructions from the control system) in the environment responsive to determining the traversability difficulty is above a traversability capability of the vehicle.

The traversability capability quantifies an ability of the vehicle to travel through environments with moisture. As described herein, a higher traversability capability indicates the vehicle can traverse more difficult terrain. The traversability capability may have a same unit of measurement or be on a same scale as the traversability difficulty so that the values can be directly compared. The traversability capability of the vehicle may be based on operational parameters of the vehicle (e.g., speed and torque) and characteristics that may affect the vehicle's ability to traverse terrain. Example vehicle characteristics include a wheel type (e.g., wheel or tracks), a wheel size, a tread type, an engine/motor type, a drive type (e.g., front, rear, or all-while drive), a make, a model, a weight, a fuel level, a tank level for a sprayer, a treatment mechanism, or coupling mechanism of the vehicle. Values of these characteristics may be determined from sensors of the vehicle. One or more of the characteristics may be variable. For example, the weight of the vehicle changes over time as the vehicle consumes fuel or applies treatments to the environment (e.g., by emptying a cargo bed or tank). Thus, the traversability capability may be a fixed value or, in some embodiments, a variable quantity that changes over time based on the real time operational parameters and characteristics of the vehicle. In some embodiments, the traversability capability is specific to one or more actions performed by the vehicle as it moves through the environment. For example, a construction vehicle may have a first traversability capability if it is performing a first construction action (e.g., moving while excavating) and a second traversability capability if it is performing a second construction action (e.g., moving without excavating).

An action in the context of step440is an action performed by the vehicle (e.g., via the control system) and intended to prevent or reduce the likelihood of the vehicle traversing an untraversable environment portion. In some cases, the action modifies (e.g., cancels) an action already being performed by the vehicle. An example action includes modifying an operational parameter of the vehicle. Modifying an operational parameter may increase the traversability capability of the vehicle, such as increasing the speed, switching to all-wheel drive, switching from speed-control to torque-control on the drive wheel motors, ceasing to apply power to the wheels so that the vehicle ‘coasts’ through the environment portion, or raising a treatment mechanism, so that the traversability capability is no longer below the traversability difficulty. Another example of an action includes sending a warning notification to an operator of the vehicle. In another example, the action modifies (e.g., cancels) work performed by the vehicle (e.g., an action applied to the environment). For example, if a construction vehicle is performing a construction action (e.g., plowing soil) that limits the speed of the vehicle (e.g., so it will not have enough speed to traverse the environment portion), the action may cease the construction action (e.g., raise the plowing tool) so the vehicle can increase its speed. In another example, if a forestry vehicle is carrying a tree, the vehicle may drop the tree to increase its speed to increase the likelihood of traversing the environment portion.

In some embodiments, the action modifies the vehicle's route such that it does not move through (or ceases to move through) the portion of the environment including moisture (e.g., see description with respect toFIG.4). If the route is modified, it may be modified so that the vehicle will move through the environment portion at a later point in time or so that the vehicle will move through the environment from a different direction (e.g., modifying the route so that the vehicle moves through the portion while traveling downhill instead of uphill). This may allow time for the conditions at the environment portion (e.g., the moisture level) to change. This may also provide the vehicle time or the opportunity to adjust one or more characteristics so it can move through the environment portion. For example, the route is modified so that the vehicle moves through the portion with a lighter machine load, such as waiting until the volume in a fuel tank or cargo bed has decreased. In another example, the vehicle modifies its weight or weight distribution by using counterweight brackets, shedding unused components, dumping material or transferring material to a storage tank, swapping components like wheels or tracks, disengaging a treatment mechanism (e.g., an excavator tool), or by attaching accessories like skids, skis, or additional idler wheels. In another example, the wheel or track width of the vehicle is modified. In some embodiments, the vehicle, another vehicle (e.g., a helper machine), or an operator may modify the environment portion (e.g., by applying sand, laying skids or boards onto the path, or blasting the area with air.)

The traversability difficulty may be determined while the vehicle is moving towards the portion of the environment. However, the traversability difficulty may be determined prior to this. For example, an image sensor (e.g., on a scout, drone, aerial imager, or satellite that is physically separate from the vehicle) captures an image of the environment portion of the environment and the traversability model is applied to the image (e.g., using cloud processing) prior to the vehicle moving through the environment. When it is time to move in the environment (e.g., later in the day or on another day), action instructions may be provided to the vehicle. Said differently, the traversability difficulty may be determined at a first time and the vehicle may perform the action based on the traversability difficulty (and the measure of traversability) at a second time, where the second time can occur at any time after the first time. In some embodiments, if the control system determines a traversability difficulty for one or more portions of the environment before the vehicle moves in the environment, the control system may determine the route based on the determined traversability difficulties (and the traversability capability of the vehicle).

As stated above, a traversability difficulty for a portion of the environment may be determined prior to the vehicle moving through the environment portion. However, the traversability difficulty may be determined or updated as the vehicle moves through the environment portion. Because the vehicle is closer to the portion of the environment, the updated traversability difficulty may be more accurate than the previously determined traversability difficulty. For example, a closer view of a body of moisture results in a more accurate determination of the size of the body, and thus, a more accurate traversability difficulty determination. If a traversability difficulty was previously determined for a portion of the environment, the vehicle may move through the portion of the environment if the traversability difficulty was not above the traversability capability of the vehicle. Below is an example description of updating the traversability difficulty for a vehicle that is traveling through the environment portion. The description is in the context ofFIG.4.

Responsive to the vehicle moving through the environment portion, the vehicle accesses a second image of the portion of the environment from a second image sensor. The image includes a second group of pixels that indicate an updated moisture level of the portion of the environment. The control system applies the traversability model to the second image. The traversability model determines the updated moisture level of the portion of the environment using the second group of pixels and determines an updated traversability difficulty for the portion of the environment using the updated moisture level. In some embodiments, the control system applies a model (e.g., to images captured by side sensors) to examine a previous environment portion (that the vehicle moved through) and a future environment portion (e.g., along a route) to determine if there is a difference in moisture or traversability difficulty. Responsive to a difference between the traversability difficulty and the updated traversability difficulty being greater than a threshold, the vehicle performs a second action.

The second image sensor may be the same image sensor that captured the first image. Alternatively, it may be a different image sensor. For example, the vehicle includes two image sensors. The first image sensor has a field of view that captures an environment portion that the vehicle is moving towards, where images from the first image sensor are used to determine a traversability difficulty for the environment portion. The second image sensor has a field of view that captures a current environment portion that the vehicle is moving through, where images from the second image sensor are used to determine a traversability difficulty for the current environment portion. In some embodiments, the second image sensor is positioned to include a view of the vehicle. For example, the second image sensor captures a view of a wheel of the vehicle in contact with the ground (e.g., to detect the presence of mud build up). In this example, an increase in wheel diameter may indicate the presence of mud build up on the tire and a decrease in wheel diameter may indicate the wheel is slipping. In some embodiments, the second image sensor is positioned to view the environment behind the vehicle (e.g., to capture the depth of track marks left by the vehicle).

The updated moisture level and updated traversability difficulty may be determined using one or more factors described with reference to step430. However, now that the vehicle is traveling (or has traveled) through the portion of the environment, the vehicle may have access to new data that can additionally or alternatively be used to determine the updated moisture level and updated traversability difficulty. For example, the control system records diagnostic information from one or more diagnostic sensors of the vehicle, where the diagnostic information may indicate an updated moisture level or traversability difficulty. For example, a height sensor is mounted to the vehicle at a known height, and information from the height sensor indicates how deep the vehicle has sunk into the ground. In another example, a hygrometer mounted to the vehicle provides humidity information. In another example, information from a level sensor or altimeter may be used to determine the gradient of the environment portion. Other example sensors of the vehicle include motion sensors such as inertial measurement units (IMUs) (e.g., to measure cab vibrations of the vehicle), GPS sensors, torque/force sensors, thermal sensors, and draft/load sensors (e.g., on a pin of a chisel plow). Due to the presence of this new data, the traversability model may include a first model to determine the traversability difficulty and a second model to determine the updated traversability difficulty. In some embodiments, if an updated traversability difficulty (or updated moisture level) for an environment portion is significantly different than the traversability difficulty (or moisture level) for the environment portion, an updated traversability difficulty (or moisture level) may be determined for one or more other environment portions. This update may inform route changes for environment portions that now exceed a threshold but previously did not.

In addition to the diagnostic information, the control system may use real time operational parameters to determine an updated moisture level or traversability difficulty. For example, if the orientation of the vehicle is unresponsive or responds slower than expected to changes in the wheel steering direction, this may indicate an increase in the moisture level or traversability difficulty of the environment portion. In another example, if the engine/motor power usage is increasing (e.g., due to mud build up), this may indicate an increase in the moisture level or traversability difficulty.

Other examples of operational parameters include a gear setting, speed, engine/motor power, engine/motor torque, and engine/motor RPM (revolutions per minute). If the operational parameters are not inherently known, they may be determined using diagnostic information from one or more sensors in the vehicle. For example, the control system determines wheel or track slip of the vehicle. Slip may be determined by comparing diagnostic information from several sensors, such as rotary encoders in the wheels, GPS, and ground-facing radar. In another example, the control system monitors control errors. If tracking errors are higher than expected or if the tracking stability is worse than expected (e.g., increased overshoot or settling time), the control system may determine that an environment portion includes a higher moisture level or traversability difficulty. In some embodiments, actions performed to the environment by the vehicle may provide an indication of a moisture level or traversability difficulty. For example, if a farming vehicle performs a treatment action, such as spraying something on the soil, differences in how the spray looks on the soil may provide an indication of a moisture level. In another example, if a construction vehicle is a bulldozer clearing soil, movement of the soil may provide an indication of a moisture level.

Referring back to the updated traversability difficulty, the control system may compare the previously determined traversability difficulty with the newly determined updated traversability difficulty. If the difference between the traversability difficulty and the updated traversability difficulty is greater than a threshold, this may indicate that the previously determined traversability difficulty was inaccurate. To account for this, the vehicle may perform a second action. Similar to the actions described with reference to step440, the second action may be performed to prevent (or reduce the likelihood of) the vehicle traversing an untraversable environment portion. The second action can include any of the actions described with reference to step440.

V.B Applying the Moisture Model

FIG.6illustrates another method for operating in an environment with moisture by a vehicle (e.g., vehicle100), in accordance with one or more embodiments. One or more steps of method600may be performed from the perspective of the control system130. The method600can include greater or fewer steps than described herein. Additionally, the steps can be performed in different order, or by different components than described herein.

Similar to step410, a vehicle (e.g., vehicle100) moves610along a route in an environment (e.g., a farming, construction, mining, or forestry environment) with moisture.

The control system identifies620an action to perform by the vehicle at a portion of the environment (as previously described, a portion is a subsection of the environment that is large enough to include one or more bodies of moisture, which are large enough for the vehicle to potentially get stuck or large enough that the vehicle can potentially damage the environment if it moves through the portion). The control system may identify the action in response to analyzing an image from an image sensor (e.g., sensor210) or analyzing diagnostic information from sensors of the vehicle. The control system may also identify the action based on instructions from an operator. For example, an operator may instruct the vehicle (e.g., a construction vehicle) to use an excavator tool to backfill a hole. The control system typically identifies the action prior to the vehicle moving through the environment portion, but it may identify the action as the vehicle is moving through the environment portion.

In the context of step620, an action is an action the vehicle may perform while in the portion of the environment (e.g., while moving through the environment portion). Examples of actions include performing a treatment action, modifying a treatment parameter, modifying an operational parameter, and modifying a sensor parameter. The identified action may be an action described with reference step440.

The control system determines630a measure of moisture (also referred to as the moisture level) for a portion of the environment by applying a moisture model to an image of the portion of the environment.

As described with reference to step420, the image of the portion of the environment includes a group of pixels that indicate a measure of moisture of the environment portion. One or more image sensors may capture the image. Example image sensors that can capture the image are described with reference to the detection mechanism110. The image sensors may be coupled to the vehicle and oriented to capture the portion of the environment.

Similar to the traversability model, the moisture model determines a measure of moisture by analyzing one or more groups of pixels in the image. For example, visual properties such as texture, reflection, and saturation indicate the presence, location, and amount of moisture. In addition to analyzing pixels of the image, the traversability model may receive non-visual information to determine the moisture level of the portion of the environment, such as temperature, humidity, wind, weather data, topography, and soil maps. For more information on determining a measure or moisture, see the above description with reference to step430. In some embodiments, the moisture model is part of the traversability model.

The control system determines640a likelihood that the vehicle performing the identified action will damage the portion of the environment based on the identified action and the determined measure of moisture for the portion of the environment.

The likelihood that the vehicle performing the identified action will damage the portion of the environment may refer to a specific type of damage or an amount of damage. An operator of the vehicle may specify the type and amount or these quantities may be predetermined. In one example, an operator specifies that they can tolerate an action slightly modifying an environment but do not want an action to form a new water run-off channel in the environment. Additional examples of damage include enlarging a preexisting water run-off channel above a threshold amount, enlarging a local depression above a threshold amount (this may increase the size of a body in the future), changing the gradient of the environment portion above a threshold amount, and compacting the ground (e.g., soil) above a threshold amount.

In a construction or forestry context, another example is an operator specifying that they can tolerate vehicles deforming a (e.g., muddy) road by a threshold amount but not more. In another example in a construction context, an operator specifies that a construction action should not result in the vehicle contacting a structure or other vehicle at the construction site. In another example, for a heavy construction vehicle with treads, an operator may specify that they can tolerate the treads damaging a first area of a construction site but not areas outside of the first area.

In a farming context, another example is an operator of a farming vehicle specifying that they can tolerate an action damaging (or killing) a few plants but do not want an action to damage (or kill) a threshold number of plants. Damage to a plant may be caused by the vehicle running it over, a component of the vehicle hitting it, or mud thrown by the vehicle hitting it. Determining whether an action will damage a plant may be based on the type of plant, a growth stage, a size, a location, or a planting configuration of the plant. Additional example types of damage to a portion of the environment include damaging a threshold number of rows in the environment and changing an irrigation pathway above a threshold amount. Another form of damage is unwanted biological consequences stemming from an action being performed in the presence of moisture. For example, planting into wet ground may be undesirable. Or applications of certain herbicides may be more/less effective if the crop is wet. In another example, environment modifications from ruts can reduce or impact the ability to harvest a crop later.

The control system may use a damage model to determine the likelihood. Generally, a higher level of moisture at the environment portion results in a higher likelihood that an action will damage the environment portion (and vice versa), however the relationship depends on the action, may not be linear, and may be based on other factors, some of which are further described below. For example, the damage model is a weighted model with a weight for each factor, where each weight indicates, for the determined measure of moisture, how strongly its corresponding factor affects the likelihood. In some embodiments, the relationship between the moisture level, the action, and the likelihood is machine learned, for example historical action data. Historical action data may include actions performed at environment portions, measures of moisture of the environment portions, and damage (if any) that the actions caused to the environment portions. In some embodiments, the damage model is trained for the specific environment that the action is performed in (e.g., agricultural fields, constructions sites, or forestry environments). Additionally, or alternatively, the damage model is trained for actions performed by specific types of vehicles. For example, the model may be trained differently for an action performed by a construction vehicle vs. a farming vehicle. Other factors that may affect the likelihood determination include:

(1) The route of the vehicle. The direction of travel through the environment portion may affect whether the action damages the environment portion. For example, a vehicle performing an action while moving uphill may be more likely to damage the environment portion than the vehicle performing the action while moving downhill. In another example, the direction of travel relative a body of moisture may determine whether the action damages the environment portion.

(2) Ground (e.g., soil) type of the environment portion. A ground type may affect how the ground responds to the action. For example, wet or damp cement will respond differently than cement that is already dry.

(3) The gradient of the environment portion. Generally, a higher gradient (e.g., regardless of the route) increases the likelihood the action will damage the environment portion while a smaller gradient decreases the likelihood. To determine the gradient, the control system may identify local minimums or maximums in the environment portion.

(4) Operational parameters. For example, a vehicle with a higher speed may increase the likelihood of the action damaging the environment portion. If the damage model determines the likelihood prior to the vehicle moving through the environment portion, the damage model may assume that the operational parameters will remain constant (or within a threshold range) while the vehicle moves through the environment portion.

(5) Characteristics of the vehicle. For example, a heavy vehicle may have a high likelihood of compacting the ground (e.g., soil) and enlarging depressions in the environment. Examples characteristics, such as wheel type, wheel size, etc., are described with reference to step440.

In some cases, the likelihood is based on the vehicle actively performing the identified action in the environment portion. These cases may occur if an initial likelihood of damaging the environment is small or if the amount of potential damage is small. In these embodiments, the control system may determine whether the action being performed is damaging the environment. For example, the control system analyzes images of the vehicle performing the action.

If the likelihood does not exceed a threshold likelihood (e.g., provided by an operator or predetermined), the identified action may be performed, for example, when the vehicle enters the environment portion. However, if the likelihood exceeds the threshold likelihood, the control system performs650a second action, where the likelihood that the vehicle performing the second action will damage the portion of the environment is less than the threshold likelihood.

An action in the context of step650is an action performed by the vehicle and intended to prevent or reduce the likelihood of the vehicle damaging the portion of environment. The second action may be one or more of the actions described with reference to step620, however the second action is either a different action or a same action that is performed with different parameters (e.g., the second action has a different tilling depth) than the identified action in step620. Depending on the situation, the second action may be performed instead of the identified action, or the second action may modify the identified action (e.g., to reduce the likelihood that the identified action will damage the environment). In another example, the second action nullifies the identified action such that the identified action is not performed (or no longer performed) by the vehicle. For example, if the control system determines that moisture (e.g., a puddle) will spread a spray treatment applied to a plant (e.g., weed) to another plant (e.g., a crop), the second action may cancel the spray treatment action being performed by the vehicle. In another example, if the control system determines that an excavator action by a construction vehicle will form a new water run-off channel that is undesirable, the second action may cancel the excavator action.

As stated above, the control system may determine the measure of moisture and the likelihoods of the first and second actions while the vehicle is moving towards or through the portion of the environment. However, the control system may determine one or more of these values prior to this. For example, the control system applies the moisture model to the image of the environment portion (e.g., using cloud processing) prior to the vehicle moving through the environment. When it is time to move in the environment (e.g., later in the day or on another day), action instructions may be provided to the vehicle. Said differently, the measure of moisture and the likelihoods of the first and second actions may be determined at a first time and the vehicle may perform the second action at a second time, where the second time can occur at any time after the first time.

In some embodiments, as the vehicle gets closer to the environment portion or travels through the environment portion, it determines an updated likelihood of the identified action damaging the portion of the environment. If the updated likelihood is below the threshold likelihood, the vehicle may perform the identified action.

Methods 400 and 600 may be performed independently. In some embodiments, the methods are interconnected. For example, the action in step440may be the identified action in step620.

V.C Implementation of Moisture Model

There are several methods to determine a measure of moisture in a captured image. One method of determining moisture information from a captured image is a moisture model that operates on a convex hull optimization model. Another method of determining moisture information from a captured image is a moisture model that operates on a fully convolutional encoder-decoder network. For example, the moisture model can be implemented as functions in a neural network trained to determine moisture information from visual information encoded as pixels in an image. The moisture model may function similarly to a pixelwise semantic segmentation model where the classes for labelling bodies of moisture indicate measures of moisture.

Herein, the encoder-decoder network may be implemented by a control system130as a moisture model705. The control system130can execute the moisture model705to identify moisture associated with pixels in an accessed image700and quickly generate an accurate measure of moisture760. To illustrate,FIG.7is a representation of a moisture model, in accordance with one example embodiment.

In the illustrated embodiment, the moisture model705is a convolutional neural network model with layers of nodes, in which values at nodes of a current layer are a transformation of values at nodes of a previous layer. A transformation in the model705is determined through a set of weights and parameters connecting the current layer and the previous layer. For example, as shown inFIG.7, the example model705includes five layers of nodes: layers710,720,730,740, and750. The control system130applies the function W1to transform from layer710to layer720, applies the function W2to transform from layer720to layer730, applies the function W3to transform from layer730to layer740, and applies the function W4to transform from layer740to layer750. In some examples, the transformation can also be determined through a set of weights and parameters used to transform between previous layers in the model. For example, the transformation W4from layer740to layer750can be based on parameters used to accomplish the transformation W1from layer710to720.

In an example process, the control system130inputs an accessed image700(e.g., the image inFIG.5E) to the model705and encodes the image onto the convolutional layer710. After processing by the control system130, the model705outputs a measure of moisture760decoded from the output layer750. In the identification layer730, the control system130employs the model705to identify moisture information associated with pixels in the accessed image700. The moisture information may be indicative of amounts of moisture at a portion of the environment and their locations in the accessed image700. The control system130reduces the dimensionality of the convolutional layer710to that of the identification layer730to identify moisture information in the accessed image pixels, and then increases the dimensionality of the identification layer730to generate a measure of moisture760. In some examples, the moisture model705can group pixels in an accessed image700based on moisture information identified in the identification layer730when generating the measure of moisture760.

As previously described, the control system130encodes an accessed image700to a convolutional layer710. In one example, a captured image is directly encoded to the convolutional layer710because the dimensionality of the convolutional layer710is the same as a pixel dimensionality (e.g., number of pixels) of the accessed image700. In other examples, the captured image can be adjusted such that the pixel dimensionality of the captured image is the same as the dimensionality of the convolutional layer710. For example, the accessed image700may be cropped, reduced, scaled, etc.

The control system130applies the model705to relate an accessed image700in the convolutional layer710to moisture information in the identification layer730. The control system130retrieves relevant information between these elements by applying a set of transformations (e.g., W1, W2, etc.) between the corresponding layers. Continuing with the example fromFIG.7, the convolutional layer710of the model705represents an accessed image700, and identification layer730of the model705represents moisture information encoded in the image. The control system130identifies moisture information corresponding to pixels in an accessed image700by applying the transformations W1and W2to the pixel values of the accessed image700in the space of convolutional layer710. The weights and parameters for the transformations may indicate relationships between the visual information contained in the accessed image and the inherent moisture information encoded in the accessed image700. For example, the weights and parameters can be a quantization of shapes, distances, obscuration, etc. associated with moisture information in an accessed image700. The control system130may learn the weights and parameters using historical user interaction data and labelled images.

In the identification layer730, the control system maps pixels in the image to associated moisture information based on the latent information about the objects represented by the visual information in the captured image. The identified moisture information can be used to generate a measure of moisture760. To generate a measure of moisture760, the control system130employs the model705and applies the transformations W3and W4to the moisture information identified in identification layer730. The transformations result in a set of nodes in the output layer750. The weights and parameters for the transformations may indicate relationships between the image pixels in the accessed image700and a measure of moisture760. In some cases, the control system130directly outputs a measure of moisture760from the nodes of the output layer750, while in other cases the control system130decodes the nodes of the output layer750into a measure of moisture760. That is, model705can include a conversion layer (not illustrated) that converts the output layer750to a measure of moisture760.

The weights and parameters for the moisture model705can be collected and trained, for example, using data collected from previously captured visual images and a labeling process. The labeling process increases the accuracy and reduces the amount of time required by the control system130employing the model705to identify moisture information associated with pixels in an image.

Additionally, the model705can include layers known as intermediate layers. Intermediate layers are those that do not correspond to convolutional layer110for the accessed image700, the identification layer730for the moisture information, and an output layer750for the measure of moisture760. For example, as shown inFIG.7, layers720are intermediate encoder layers between the convolutional layer710and the identification layer730. Layer740is an intermediate decoder layer between the identification layer730and the output layer750. Hidden layers are latent representations of different aspects of an accessed image that are not observed in the data but may govern the relationships between the elements of an image when identifying a measure of moisture associated with pixels in an image. For example, a node in the hidden layer may have strong connections (e.g., large weight values) to input values and values of nodes in an identification layer that share the commonality of moisture information. Specifically, in the example model ofFIG.7, nodes of the hidden layers720and740can link inherent visual information in the accessed image700that share common characteristics to help determine moisture information for one or more pixels.

Additionally, each intermediate layer may be a combination of functions such as, for example, residual blocks, convolutional layers, pooling operations, skip connections, concatenations, etc. Any number of intermediate encoder layers720can function to reduce the convolutional layer to the identification layer and any number of intermediate decoder layers740can function to increase the identification layer730to the output layer750. Alternatively stated, the encoder intermediate layers reduce the pixel dimensionality to the moisture identification dimensionality, and the decoder intermediate layers increase the identification dimensionality to the measure of moisture dimensionality.

Furthermore, in various embodiments, the functions of the model705can reduce the accessed image700and identify any number of objects in an environment. The identified objects are represented in the identification layer730as a data structure having the identification dimensionality. In various other embodiments, the identification layer can identify latent information representing other objects in the accessed image. For example, the identification layer730can identify a result of a plant treatment, soil, an obstruction, or any other object in the environment.

Other models described herein, such as the traversability model and the damage model, may also be encoder-decoder networks similar to the moisture model705illustrated inFIG.7. That is, an encoder-decoder network may be used to extract a traversability difficulty of an environment portion, or an expected damage of an action to the environment portion. In some cases, one encoder can be used for multiple decoders. For example, a single image can be encoded onto a convolutional neural network and the traversability, moisture, and damage expectations may be extracted from that image.

V.D Training a Moisture Model

The control system130or another entity may train the moisture model (e.g., moisture model250). For example, the moisture model is trained using a plurality of labelled images of one or more environment portions. The labels in the images may indicate pixels with moisture information. The labels may be designated by an operator or labeled by someone offsite. In addition to labeling an image, non-visual information, such as temperature, humidity, wind, weather data (e.g., historical rainfall), topography, or a soil map, may be associated with the labeled images and used by the control system130to train the moisture model.

As described above, training the moisture model generates functions that are able to identify latent information in an image that corresponds to moisture information. The control system130may train the moisture model using the labelled images such that the moisture model tags a captured image with one or more measures of moisture. This approach allows the vehicle or control system130to determine a measure of moisture for an environment portion.

The control system130can train the moisture model periodically during operation of the vehicle, at a determined time, or before the moisture model is implemented on a vehicle. Additionally, the moisture model can be trained by another system such that the moisture model can be implemented on a control system of a vehicle as a standalone model. Notably, in some examples, the aspect of the control system130that trains the moisture model may not be collocated on the vehicle. That is, the moisture model may be trained on a machine separate from the vehicle100and transferred to the vehicle.

Other models described herein, such as the traversability model and the damage model, may also be trained similar to the moisture model. That is, a labeling process may be used to train the traversability model or the damage model. For example, the traversability model is trained using images that are labelled with moisture information and additional factor information, such as ground (e.g., soil) information, gradient information, or a depth of track marks in the images.

VI. Control System

FIG.8is a block diagram illustrating components of an example machine for reading and executing instructions from a machine-readable medium. Specifically,FIG.8shows a diagrammatic representation of control system130in the example form of a computer system800. The computer system800can be used to execute instructions824(e.g., program code or software) for causing the machine to perform any one or more of the methodologies (or processes) described herein. In alternative embodiments, the machine operates as a standalone device or a connected (e.g., networked) device that connects to other machines. In a networked deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

The example computer system800includes one or more processing units (generally processor802). The processor802is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a control system, a state machine, one or more application specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination of these. The computer system800also includes a main memory804. The computer system may include a storage unit816. The processor802, memory804, and the storage unit816communicate via a bus808.

In addition, the computer system800can include a static memory806, a graphics display810(e.g., to drive a plasma display panel (PDP), a liquid crystal display (LCD), or a projector). The computer system800may also include alphanumeric input device812(e.g., a keyboard), a cursor control device814(e.g., a mouse, a trackball, a joystick, a motion sensor, or other pointing instrument), a signal generation device818(e.g., a speaker), and a network interface device820, which also are configured to communicate via the bus808.

The storage unit816includes a machine-readable medium822on which is stored instructions824(e.g., software) embodying any one or more of the methodologies or functions described herein. For example, the instructions824may include the functionalities of modules of the system130described inFIG.2. The instructions824may also reside, completely or at least partially, within the main memory804or within the processor802(e.g., within a processor's cache memory) during execution thereof by the computer system800, the main memory804and the processor802also constituting machine-readable media. The instructions824may be transmitted or received over a network826via the network interface device820.

VII. Additional Considerations

In the description above, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the illustrated system and its operations. It will be apparent, however, to one skilled in the art that the system can be operated without these specific details. In other instances, structures and devices are shown in block diagram form in order to avoid obscuring the system.

Some portions of the detailed descriptions are presented in terms of algorithms or models and symbolic representations of operations on data bits within a computer memory. An algorithm is here, and generally, conceived to be steps leading to a desired result. The steps are those requiring physical transformations or manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

Some of the operations described herein are performed by a computer physically mounted within a machine. This computer may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of non-transitory computer readable storage medium suitable for storing electronic instructions.

The figures and the description above relate to various embodiments by way of illustration only. It should be noted that from the following discussion, alternative embodiments of the structures and methods disclosed herein will be readily recognized as viable alternatives that may be employed without departing from the principles of what is claimed.

Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for a system and a process for operating a vehicle in an environment with moisture including a control system executing a semantic segmentation model. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those, skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.