Remote control operation of a machine using a light beam for depth perception

A system may include a light source configured to emit a light beam. The system may further include a camera configured to obtain image data of an environment that includes the light beam and a target object. The system may further include a wireless communication component configured to provide the image data to a remote control device that controls an operation of a machine. The image data may be provided to cause the image data to be included in a video feed, of the environment, displayed by the remote control device, The light source may be configured to cause the light beam, in the video feed, to provide a visual indication of a distance from the machine to the target object. Control of the operation of the machine, through the remote control device, is based, at least in part, on the light beam displayed in the video feed.

TECHNICAL FIELD

The present disclosure relates generally to a remote control operation of a machine and, for example, to a remote control operation using a light beam for depth perception.

BACKGROUND

A machine can be controlled using a remote control device. The remote control device may operate the construction machine without line-of-sight with respect to the machine. Accordingly, the machine may include sensor devices that generate sensor data that provides a measure of depth perception (or perception of depth) with respect to an environment surrounding the machine. The remote control device may use the sensor data to control the machine based on the depth perception provided by the sensor data.

In some instances, the environment may be subject to harsh environmental conditions. In this regard, the harsh environmental conditions may cause the sensor devices to prematurely experience a failure. As a result of the sensor devices prematurely experiencing a failure, the machine may be operated, by the remote control device, remotely without line-of-sight and without depth perception.

Operating the machine without line-of-sight and without depth perception causes the machine to operate inefficiently (e.g., inefficient use of components such as an engine, an implement, among other examples), may cause damage to the machine (e.g., due to unascertained conditions of the environment and/or a terrain in the environment), may cause a task to be performed incorrectly by the machine, among other examples.

SUMMARY

A system for remote control of a machine includes a light source configured to emit a light beam; a camera configured to obtain image data of an environment that includes the light beam and an object, wherein the light beam is configured to provide a visual indication of a distance from the machine to the object; and a controller configured to cause the image data to be provided to a remote control device that controls an operation of the machine, wherein the image data is provided to cause the image data to be included in a video feed, of the environment, displayed by the remote control device, wherein control of the operation of the machine, through the remote control device, is based, at least in part, on the light beam displayed in the video feed, and wherein a visual characteristic of the light beam, in the video feed, changes to indicate a change in the distance from the machine to the object.

A machine includes an implement; a light source configured to emit a light beam; a camera configured to obtain image data of an environment that includes the light beam and a target object; a wireless communication component configured to communicate with a remote control device that controls an operation of the machine; and a controller configured to cause the wireless communication component to provide the image data to the remote control device, wherein the image data is provided to cause the image data to be included in a video feed, of the environment, displayed by the remote control device, wherein the light source is configured to cause the light beam, in the video feed, to provide a visual indication of a distance from the machine to the target object, and wherein control of the operation of the machine, through the remote control device, is based, at least in part, on the distance indicated by the light beam displayed in the video feed.

A system includes a light source configured to emit a light beam; a camera configured to obtain image data of an environment that includes the light beam and a target object; and a wireless communication component configured to provide the image data to a remote control device that controls an operation of a machine, wherein the image data is provided to cause the image data to be included in a video feed, of the environment, displayed by the remote control device, and wherein the light source is configured to cause the light beam, in the video feed, to provide a visual indication of a distance from the machine to the target object.

DETAILED DESCRIPTION

The present disclosure is directed to a system that uses a light beam to provide depth perception to a remote control device operating a machine. For example, a light source may be mounted to the machine. For instance, the light source may be mounted to an operator cabin of the machine, to a frame of the machine, to existing lighting fixtures of the machine, among other examples. The light source may include a laser light source, an infrared light source, among other examples. The light beam may include a laser light beam, an infrared light beam, among other examples of focused light beam.

The light beam may be projected into an environment surrounding the machine and may provide feedback to an operator of the remote control device when the light beam is distorted by objects provided in a travel path of the machine. For example, the light beam may provide a visual indication of a distance from the machine to an object. For instance, a visual characteristic of the light beam may change to indicate a change in the distance from the machine to the object. The visual characteristic of the light beam may include a shape, a size, a location, among other examples.

In some examples, the light source may be directed toward a ground surface. In this regard, the light source may be configured to cause the light beam to be projected onto the ground surface to indicate that the distance is a first distance from the machine to a target object (e.g., a dump target such as a truck bed of another machine). As the machine approaches the target object, the light beam may be projected onto the target object to indicate that the distance has decreased from the first distance to a second distance.

The light beam may be configured to gradually move vertically upward a side surface of the target object as the machine continues to approach the target object. The light source may be configured to cause the light beam to be projected onto a particular portion of the target object to indicate that the machine is at a target distance, from the target object, for performing a dumping operation. The target distance may be an optimal distance for efficiently and properly performing the dumping operation to dump a load, carried by the machine, onto the target object (e.g., into the truck bed). The machine may continue to approach the target object until the light beam is projected onto the particular portion of the target object.

The target distance may be different for different target objects (e.g., based on different sizes of the target objects, based on different operations associated with the target objects, among other examples). Accordingly, the light source may be configured to cause the light beam to be projected onto different portions of different target objects to indicate different target distances.

The machine may include a camera configured to obtain image data of the environment that includes the light beam and the target object. The image data may be provided to the remote control device to cause the image data to be included in a video feed, of the environment, displayed by the remote control device. The image data, of the light beam in the video feed, may provide a visual indication of the distance. The visual indication may be used by the remote control device to control the machine.

In some instances, the machine may include multiple light sources. Each light source may emit a light beam that is projected into the environment. The multiple light beams may provide visual indications, in the video feed, of the distance from the machine to the target object. In some examples, the multiple light sources may be configured to cause an intersection of the multiple beams to indicate the target distance. In this regard, the machine may approach the target object until the intersection is projected onto a desired portion of the target object.

While examples herein may be directed to a dumping operation, the present disclosure is applicable to other types of operations such as a digging operation, a navigating operation, among other examples. For example, the light source may be configured to provide a visual indication of a target distance, from the machine, for performing the digging operation.

The term “machine” may refer to a device that performs an operation associated with an industry such as, for example, mining, construction, farming, transportation, or another industry. Moreover, one or more implements may be connected to the machine. As an example, a machine may include a construction vehicle, a work vehicle, or a similar vehicle associated with the industries described above.

FIG.1is a diagram of an example implementation100described herein. As shown inFIG.1, example implementation100includes a machine105and a remote control device180. Machine105is embodied as an earth moving machine, such as a wheel loader. Alternatively, machine105may be another type of machine, such as an excavator, a dozer, among other examples.

As shown inFIG.1, machine105includes ground engaging members110, an engine115, an operator cabin120, a wireless communication component125, a stick130, and a machine work tool135. Ground engaging members110may include wheels (as shown inFIG.1), tracks, rollers, among other examples, for propelling machine105. Ground engaging members110are mounted on a machine body and are driven by engine115and drive trains (not shown). Engine115may be operatively connected to provide power to drive at least one of the ground engaging members110. Operator cabin120is supported by the machine body. Operator cabin120may include an integrated display (not shown) and operator controls (not shown), such as, for example, an integrated joystick. The operator controls may include one or more input components.

For an autonomous machine, the operator controls may not be designed for use by an operator and, rather, may be designed to operate independently from an operator. In this case, for example, the operator controls may include one or more input components that provide an input signal for use by another component without any operator input.

Wireless communication component125may include one or more devices that are capable of communicating with remote control device180, as described herein. Wireless communication component125may include a transceiver, a separate transmitter and receiver, an antenna, among other examples. Wireless communication component125may communicate with the one or more machines using a short-range wireless communication protocol such as, for example, BLUETOOTH® Low-Energy, BLUETOOTH®, Wi-Fi, near-field communication (NFC), Z-Wave, ZigBee, or Institute of Electrical and Electronics Engineers (IEEE) 802.154, among other examples.

Additionally, or alternatively, wireless communication component125may communicate with remote control device180, one or more other machines and/or one or more devices via a network that includes one or more wired and/or wireless networks.

Stick130is pivotally mounted at its proximal end to the machine body and is articulated relative to the machine body by one or more fluid actuation cylinders (e.g., hydraulic or pneumatic cylinders), electric motors, and/or other electro-mechanical components. Stick130may be referred to as a linkage. Machine work tool135is mounted at a distal end of stick130and may be articulated relative to stick130by one or more fluid actuation cylinders, electric motors, and/or other electro-mechanical components. Machine work tool135may be a bucket (as shown inFIG.1) or another type of tool or implement that may be mounted on stick130. Machine work tool135may be referred to as an implement.

Machine105may include a light source140, a light beam145, a camera150, a controller155(e.g., an electronic control module (ECM), a computer vision controller, an autonomy controller, among other examples), one or more inertial measurement units (IMUs)160(referred to herein individually as “IMU160,” and collectively referred to as “IMUs160”), and a frame165.

Light source140may include a laser light source, an infrared light source, among other examples of devices that emit focused light beams. As shown inFIG.1, light source140may be mounted on operator cabin120. Alternatively, light source140may be mounted on other portions of machine105, such as the machine body, a frame, a chassis, among other examples.

Light source140may be configured to emit light beam145. In some examples, light source140may be configured to cause light beam145to be projected onto ground surface170(e.g., projected to a front side of machine105). Additionally, or alternatively, light source140may be configured to cause light beam145to be projected in a manner that causes light beam145to be parallel to ground surface170. Additionally, or alternatively, light source140may be configured to cause light beam145to be projected to a lateral side of machine105. Additionally, or alternatively, light source140may be configured to cause light beam145to be projected to a rear portion of machine105.

Light beam145may include a laser light beam, an infrared light beam, among other examples of focused light beams. Light beam145may be distorted by objects (e.g., target objects) located in a traveling path of machine105. Light beam145may provide an indication of a distance from machine105to a target object. For example, a visual characteristic of light beam145may change to indicate a distance from machine105to a target object and to indicate a change in the distance from machine105to the target object. The visual characteristic of light beam145may include a shape, a size, a location, among other examples.

With respect the visual characteristic changing, light beam145may be a first shape when the distance from machine105to the target object is a first distance, a second shape when the distance from machine105to the target object is a second distance that is less than the first distance, and so on. Additionally, or alternatively, light beam145may be a first size (e.g., a first width) when the distance from machine105to the target object is the first distance, a second size (e.g., a second width) that is less than the first size when the distance from machine105to the target object is the second distance, and so on.

Additionally, or alternatively, light beam145may be projected onto ground surface170when the distance from machine105to the target object is the first distance, projected onto a bottom portion of the target object when the distance from machine105to the target object is the second distance, and so on. The distances and the visual characteristic are merely provided as examples. In practice, other examples may be used for the distances and the visual characteristic.

Camera150may include one or more devices that are capable of obtaining and providing image data of an environment (that includes light beam145and the target object) surrounding machine105. The image data may be provided to remote control device180to cause the image data to be included in a video feed, of the environment, displayed by remote control device180.

In some implementations, camera150may be a monocular camera and the image data may be two-dimensional (2D) image data. Alternatively, camera150may be a stereo camera and the image data may be three-dimensional (3D) image data. In some situations, camera150may be configured to detect infrared light beams. For example, infrared filters may be removed from camera150to enable camera150to detect infrared light beams. In this regard, the image data may include data regarding the infrared light beams detected by camera150.

Controller155may control and/or monitor operations of machine105. For example, controller155may control and/or monitor the operations of machine105based on signals from light source140, light beam145, signals from camera150, signals from IMUs160and/or signals from remote control device180. In some examples, controller155may cause the image data to be provided to remote control device180. For example, controller155may cause wireless communication component125to provide the image data to be provided to remote control device180to cause the image data to be included in the video feed of the environment.

As shown inFIG.1, IMUs160are installed at different positions on components or portions of machine105, such as, for example, stick130, the machine body, engine115, among other examples. An IMU160includes one or more devices that are capable of receiving, generating, storing, processing, and/or providing signals indicating a position and orientation of a component, of machine105, on which IMU160is installed. For example, IMU160may include one or more accelerometers and/or one or more gyroscopes.

The one or more accelerometers and/or the one or more gyroscopes generate and provide signals that can be used to determine a position and orientation of the IMU160relative to a frame of reference and, accordingly, a position and orientation of the component. While the example discussed herein refers to IMUs160, the present disclosure is applicable to using one or more other types of sensor devices that may be used to determine a position and orientation of a component of machine105.

Remote control device180may include one or more devices that are configured to be used for a remote control operation of machine105(e.g., a remote control operation without line-of-sight with respect to machine105). For example, remote control device180may include one or more displays, one or more operator controls (similar to the operator controls of machine105), one or more controllers (similar to controller155), a wireless communication component (similar to wireless communication component125), among other examples.

Remote control device180may establish a communication with machine105via the wireless communication component and may control machine105using the wireless communication. Remote control device180may display, via the one or more displays, the video feed (including the image data obtained by camera165). In some examples, remote control device180may include one or more input components (e.g., a keyboard, a microphone, joysticks, buttons, pedals, among other examples) that are used to provide input regarding the video feed.

Remote control device180may control an operation of machine105based on light beam145displayed in the video feed. As an example, remote control device180may control the implement of machine105during a dumping operation involving the target object (e.g., a dump target). The image data may be provided to remote control device180to facilitate control of the implement during the dumping operation. Light beam145, in the video feed, may be configured to provide a visual indication of the distance from machine105to the target object. The visual indication of light beam145and the change of the visual characteristic of light beam145may be used to facilitate control of the implement during the dumping operation.

In some implementations, the visual characteristic, of light beam145, may be configured to change based on a velocity of machine105. For example, a rate of change of light beam145may be based on the velocity of machine105. In this regard, the rate of change of light beam145may provide an indication of the velocity of machine105.

As indicated above,FIG.1is provided as an example. Other examples may differ from what was described in connection withFIG.1.

FIG.2is a diagram of an example implementation200using a light source described herein. As shown inFIG.2, example implementation200may include machine105and target object210. Some of the elements ofFIG.2have been described above in connection withFIG.1. Target object210may be another machine and/or a portion of the other machine. As shown inFIG.2, target object210may be a truck bed of a dump truck. Target object210may include a top portion215and a bottom portion220.

In some implementations, light source140may be configured to cause light beam145to provide an indication that machine105is located at a target distance from the dump target for performing a dumping operation. The target distance may be an optimal distance for efficiently and properly performing the dumping operation to dump a load, carried by the machine, onto target object210.

As an example, light source140may be configured to cause light beam145to be projected onto a portion of the dump target (e.g., top portion215) when machine105is located at the target distance. Alternatively, light source140may be configured to cause light beam145to be projected onto ground surface170when machine105is located at a distance from target object210that exceeds the target distance.

As shown inFIG.2, light beam145is projected onto ground surface170. Light beam145may be projected onto ground surface170to indicate that machine105is located at a first distance, from target object210, that exceeds the target distance.

As indicated above,FIG.2is provided as an example. Other examples may differ from what was described in connection withFIG.2.

FIG.3is a diagram of an example implementation300using a light source described herein. As shown inFIG.3, example implementation300may include machine105and target object210. Some of the elements ofFIG.3have been described above in connection withFIG.1andFIG.2.

As shown inFIG.3, light beam145is projected onto bottom portion220of target object210. Light beam145may be projected onto bottom portion220to indicate that a distance between machine105and target object210has decreased from the first distance to a second distance.

As indicated above,FIG.3is provided as an example. Other examples may differ from what was described in connection withFIG.3.

FIG.4is a diagram of an example implementation400using a light source described herein. As shown inFIG.4, example implementation400may include machine105and target object210. Some of the elements ofFIG.4have been described above in connection withFIGS.1-3.

As shown inFIG.4, light beam145is projected onto top portion215of target object210. In this regard, light beam145may be configured to gradually move vertically upward a side surface of target object210as machine105continues to approach target object210. Light beam145may be projected onto top portion215to indicate that the distance between machine105and target object210has decreased from the second distance to a third distance (e.g., the target distance).

As indicated above,FIG.4is provided as an example. Other examples may differ from what was described in connection withFIG.4.

FIG.5is a diagram of an example implementation500using multiple light sources described herein. As shown inFIG.5, example implementation500may include machine105and a target object510(e.g., a dump target, such as a truck bed of a dump truck). Machine105may include multiple light sources140, namely first light source140-1, second light source140-2, and third light source140-3. As shown inFIG.5, light sources140may be mounted on operator cabin120. Second light source140-2may be provided between first light source140-1and third light source140-3.

Light sources140may emit multiple light beams145. For example, first light source140-1may emit first light beam145-1, second light source140-2may emit second light beam145-2, and third light source140-3may emit third light beam145-3. Light beams145may be parallel to ground surface170and may be projected onto a side surface of target object510.

Light beams145may be configured such that distances between light beams145indicate whether machine105is approaching an object provided in a path that is perpendicular to a travel path of machine105. For example, light beams145may indicate that machine105is approaching an object that is provided in a path that is perpendicular to the travel path of machine105when a distance DLB12(between first light beam145-1and second light beam145-2) is equal to DLB23(between second light beam145-2and third light beam145-3).

As shown inFIG.5, distance DLB12is equal to DLB23. In this regard, distance DLB12and distance DLB23may indicate (to the operator of remote control device180) that target object510is provided in a path that is perpendicular to the travel path of machine105.

In some situations, distance DLB12and distance DLB23may be configured to change based on the velocity of machine105. For example, a rate of change of distance DLB12and distance DLB23may be based on the velocity of machine105.

In some instances, light sources140may be configured to cause an intersection of light beams145to indicate a target distance from machine105. The target distance may be an optimal distance for performing the dumping operation to dump a load, carried by machine105, onto target object510. As machine105approaches target object510and as a distance between machine105and target object510approaches the target distance, distance DLB12and DLB23may decrease.

The intersection may be projected onto target object510to indicate that machine105has reached the target distance. Machine105may be controlled to initiate the dumping operation when machine105reaches the target distance. In some situations, light sources140may be configured to cause the intersection to be located at a particular distance Di from a front portion of the implement. The front portion of the implement may be a location of furthest reach for the implement. In this regard, the particular distance Di may be selected to prevent the front portion colliding with target object510.

As indicated above,FIG.5is provided as an example. Other examples may differ from what was described in connection withFIG.5.

FIG.6is a diagram of an example implementation600using multiple light sources described herein. As shown inFIG.6, example implementation600may include machine105and target object510. Some of the elements ofFIG.6have been described above in connection withFIG.6.

As shown inFIG.6, distance DLB12is different than DLB23. In this regard, distance DLB12and distance DLB23may indicate (to the operator of remote control device180) that target object510is provided in a path that is not perpendicular to the travel path of machine105.

As indicated above,FIG.6is provided as an example. Other examples may differ from what was described in connection withFIG.6.

INDUSTRIAL APPLICABILITY

The present disclosure is directed to a system that uses a light beam to provide depth perception to a remote control device operating a machine. The light beam may be projected into an environment surrounding the machine and may provide feedback to an operator of the remote control device when the light beam is distorted by objects provided in a travel path of the machine. The light beam may provide an indication of a distance from the machine to an object. For example, a visual characteristic of the light beam may change to indicate a change in the distance from the machine to the object.

An existing machine may include sensor devices that generate sensor data providing a measure of depth perception. An existing remote control device may use the sensor data to control the existing machine based on the depth perception provided by the sensor data. Currently, the sensor devices prematurely experience a failure. The premature failure may cause the existing remote control device to operate the existing machine without depth perception.

By using the light beam to provide depth perception, the system of the present disclosure may enable the remote control device to operate the machine without being subject to premature failures of the sensor devices.

The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations may be made in light of the above disclosure or may be acquired from practice of the implementations. Furthermore, any of the implementations described herein may be combined unless the foregoing disclosure expressly provides a reason that one or more implementations cannot be combined. Even though particular combinations of features are recited in the claims and/or disclosed in the specification, these combinations are not intended to limit the disclosure of various implementations. Although each dependent claim listed below may directly depend on only one claim, the disclosure of various implementations includes each dependent claim in combination with every other claim in the claim set.

As used herein, “a,” “an,” and a “set” are intended to include one or more items, and may be used interchangeably with “one or more.” Further, as used herein, the article “the” is intended to include one or more items referenced in connection with the article “the” and may be used interchangeably with “the one or more.” Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise. Also, as used herein, the term “or” is intended to be inclusive when used in a series and may be used interchangeably with “and/or,” unless explicitly stated otherwise (e.g., if used in combination with “either” or “only one of”). Further, spatially relative terms, such as “below,” “lower,” “above,” “upper,” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. The spatially relative terms are intended to encompass different orientations of the apparatus, device, and/or element in use or operation in addition to the orientation depicted in the figures. The apparatus may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein may likewise be interpreted accordingly.