Display system for machine

A display system for displaying image data of an environment of a machine includes a display screen and a plurality of imaging devices communicably coupled to the display screen. Each of the plurality of imaging devices generates image data of the environment of the machine and has an associated operating parameter. The plurality of imaging devices stores the image data in an uncompressed form. The plurality of imaging devices compresses the image data and generates signals indicative of the compressed image data. The plurality of imaging devices receives the operating parameter associated with at least one another imaging device. The plurality of imaging devices transmits the compressed image data to the display screen based on the associated operating parameter and the received parameter of the at least one another imaging device. The plurality of imaging devices displays an image on the display screen based on the transmitted image data.

TECHNICAL FIELD

The present disclosure relates to a display system for a machine. More specifically, the present disclosure relates to the display system for displaying image data of an environment of the machine.

BACKGROUND

A machine, such as a hydraulic excavator, is a self-propelled construction machine designed for conducting work, such as excavating. The hydraulic excavator may include an upper swiveling body installed on a base having a crawler or wheel carriage. A self-propelled working machine, such as the hydraulic excavator, may be provided with a surveillance mechanism for surveying around the upper swiveling body in order to ensure the safety for working and to improve the operability of the machine is known. This surveillance mechanism may be constituted by one or more cameras attached to the upper swiveling body and a display screen installed at a position in front of an operator's seat at an operator station. Images taken by the camera may be displayed on the display screen.

In order to maximize safety while operating the machine, the plurality of cameras may be configured to obtain a widest possible viewing angle around the working machine. The plurality of cameras may generate image data corresponding to the surround-view and communicate those images to the display. However, since the plurality of cameras is installed, the image data may need to be sent to an electronic control unit (ECU) over a data link of the machine. The ECU then combines images from the plurality of cameras to provide a complete view of the environment around the machine. The above system inherently requires an expensive ECU with abundance of processing power and memory. Further, image data is compressed to communicate volumes of image data from the plurality of cameras to the ECU as the capacity of data link is limited due to system design requirements, reducing overall image quality affecting tracking of people and objects around the machine.

U.S. Pat. No. 7,787,013 (hereinafter referred to as '013 reference) describes a monitor system with plurality of cameras. The '013 reference includes a method for monitoring a monitoring area wherein each of the plurality of cameras capture an image of an image capturing area. The method includes changing an image capturing area of each of the plurality of imaging devices based on an image capturing characteristics information so as to form a gapless wide range image. However, the '013 reference does not disclose details about any solution to prevent increase in overall cost of the machine or reduction of overall image quality.

Therefore, an improved display system for displaying image data of the environment of the machine is required.

SUMMARY

In an aspect of the present disclosure, a display system for displaying image data of an environment of a machine is provided. The display system includes a display screen and a plurality of imaging devices communicably coupled to each other and the display screen. Each of the plurality of imaging devices has an associated operating parameter. Each of the plurality of imaging devices generates the image data of the environment of the machine. Each of the plurality of imaging devices stores the image data in an uncompressed form. The plurality of imaging devices compresses the image data and generates signals indicative of the compressed image data. The plurality of imaging devices receives an operating parameter associated with at least one another imaging device from the plurality of imaging devices. The plurality of imaging devices transmits the compressed image data to the display screen based on the associated operating parameter, and the received operating parameter of the at least one another imaging device. The plurality of imaging devices display an image on the display screen based on the transmitted image data.

In another aspect of the present disclosure, a method of displaying image data of an environment of a machine is disclosed. The machine includes a display screen and a plurality of imaging devices communicably coupled to the display screen. The method includes generating the image data of the environment of the machine by the plurality of imaging devices. Each of the plurality of imaging devices has an associated operating parameter. The method includes receiving an operating parameter associated with at least one another imaging device from the plurality of imaging devices. The method also includes transmitting the image data to the display screen based on the associated operating parameter and the received operating parameter of the at least one another imaging device. The method further includes displaying an image on the display screen based on the transmitted image data by the plurality of imaging devices.

In yet another aspect of the present disclosure, a machine including an imaging device is disclosed. The imaging device generates image data of an environment of the machine. The imaging device stores the image data in an uncompressed form. The imaging device compresses the image data and generates signals indicative of the compressed image data. The imaging device transmits signals indicative of the compressed image data based on the associated operating parameter, and the received operating parameter of the at least one another imaging device. The machine further includes a display screen communicably coupled to the plurality of imaging devices. The display screen receives the signals indicative of the compressed image data transmitted by the plurality of imaging devices and displays an image on the display screen based on the signals indicative of the compressed image data.

DETAILED DESCRIPTION

Wherever possible, the same reference numbers will be used throughout the drawings to refer to same or like parts.FIG. 1shows an exemplary machine100. The machine100is illustrated as a hydraulic excavator which may be used, for example, for construction and other allied industries. While the following detailed description describes an exemplary aspect in connection with the hydraulic excavator, it should be appreciated that the description applies equally to the use of the present disclosure in other machines as well.

The machine100includes an upper swiveling body102supported on a ground engaging element104. Although, the ground engaging element104is illustrated as continuous tracks, it should be contemplated that the ground engaging element104may be any other type of ground engaging element as well, for example, wheels etc. The machine100further includes a working mechanism106for conducting work, such as, for example, to excavate landsides or otherwise move material. The working mechanism106is an excavating mechanism including a boom108, an arm110, and a bucket112, which serves as a front attachment. Additionally, the upper swiveling body102includes a counterweight114provided at a tail end.

Further referring toFIG. 1, the machine100includes an operator station116coupled to the upper swiveling body102. The operator station116includes a display screen118and other levers or controls (not shown) for operating the machine100. The machine100further includes an imaging device120including a plurality of imaging devices120generating image data (not shown), the plurality of imaging devices120positioned at spaced locations on the machine100so as to capture images of varying regions around the machine100. In an embodiment, each of the image data may include a time stamp depicting a time at which the image data is being generated.

Each of the plurality of imaging devices120has an associated operating parameter. In an embodiment, the associated operating parameter may include an orientation of the plurality of imaging devices120relative to each other or the time at which the image data is generated or a quality of the image data. In some embodiments, the associated operating parameter may include both the orientation of the plurality of imaging devices120relative to each other and the quality of the image data. The illustrated orientations of the plurality of imaging devices120are described later in the disclosure. In some embodiments, each of the plurality of imaging devices120puts a time stamp in the image data being generated.

In the illustrated embodiment ofFIG. 1, the plurality of imaging devices120include a first imaging device120a, a second imaging device120b, a third imaging device120c, and a fourth imaging device120dmounted on the upper swiveling body102. The first imaging device120a, the second imaging device120b, the third imaging device120c, and the fourth imaging device120dmay be referred to together as the imaging devices120. In an embodiment, each of the plurality of imaging devices120may include cameras capable of capturing both still and moving images. In another exemplary embodiment, imaging devices120may each include smart cameras or smart vision systems having a dedicated processor on-board, including video processing acceleration provided by programmable state array (FPGA), digital signal processor (DSP), general purpose graphics processing unit (GP-GPU), or any other suitable microprocessor with supporting application software.

In the illustrated embodiment, the first imaging device120ais installed adjacent the boom108and obtains images of a right-front side RFS of an environment E of the machine100. The second imaging device120bis installed at a position substantially in middle of the counterweight114and covers a back side BS of the upper swiveling body102. Further, the third imaging device120cis installed at a position behind the operator station116and upon the upper swiveling body102for obtaining images of a left side LS, and the fourth imaging device120dis installed on a position opposite to the third imaging device120con the upper swiveling body102for obtaining images of a right side RS of the environment E of the machine100. With the provision of the first imaging device120a, the second imaging device120b, the third imaging device120c, and the fourth imaging device120d, a view of substantially all sides around the machine100is generated.

Moving on toFIG. 2, the plurality of imaging devices120further include a fifth imaging device200and a central imaging device202. In the illustrated embodiment, the fifth imaging device200is installed above the operator station116and obtains images of a left-front side LFS of the environment E of the machine100. Further, the central imaging device202is installed on a rear-side of the boom108for obtaining images of a central region of front side CFS of the machine100. It should be contemplated that the positions of the imaging devices120are merely exemplary. Any other such combination of the positions of the imaging devices120may be used which may provide a complete view of the environment E of the machine100.

The central imaging device202may be a smart camera with embedded digital processor on-board. The central imaging device202is communicably coupled to each of the plurality of imaging devices120and the display screen118. In an embodiment, the obtained images of the first imaging device120a, the fifth imaging device200, and the central imaging device202may be combined together for providing additional visibility. Further, the central imaging device202may be used for combining together the obtained images.

Referring toFIG. 3, a display system300is illustrated. The display system300includes the display screen118and the plurality of imaging devices120. The display screen118includes a display section302and a control section304. The display section302is capable of displaying digital images and can be controlled, via manual controllers306of the control section304, or via touchscreen control of the display section302, or via a combination of manual and touchscreen controllers. The display screen118is communicably coupled to the plurality of imaging devices120. In an embodiment, the display screen118is communicably coupled to the imaging devices120, the fifth imaging device200, and the central imaging device202for receiving the respective image data.

As illustrated, the display section302is divided in a first zone Z1, a second zone Z2, a third zone Z3, and a fourth zone Z4. The first zone Z1displays a first image data (not shown) generated as a first signal (not shown) from the first imaging device120a. The first image data may be produced by merging image data from the captured images of the first imaging device120a, the fifth imaging device200, and the central imaging device202. Similarly, the second zone Z2, the third zone Z3, and the fourth zone Z4may display a second image data (not shown), a third image data (not shown), and a fourth image data (not shown) received from the second imaging device120b, the third imaging device120c, and the fourth imaging device120d, respectively. The first image data, the second image data, the third image data, and the fourth image data are stored in an uncompressed form by the respective imaging devices. The central imaging device202may selectively compress the image data from the respective imaging devices120before communicating to the display screen118. The display screen118may display the respective image data in the respective zones.

FIG. 4shows a view of the environment E of the machine100captured by the fifth imaging device200, that is of the left-front side LFS of the environment E of the machine100. A first person P1is clearly visible on the left-hand side of the boom108while a second person P2is obscurely visible on the right-hand side of the boom108. In contrast,FIG. 5displays a view of the environment E of the machine100captured by the first imaging device120a, that is of the right-front side RFS of the environment E of the machine100. The second person P2is clearly visible on the right-hand side of the boom108while the first person P1is obscurely visible on the left-hand side of the boom108.

In an exemplary application, where the first person P1is being tracked based upon an object detection technology or algorithm, already known in the art, the fifth imaging device200may hand-off all tracking characteristics associated with the first person P1to the first imaging device120awhen the first person P1is about to enter or enters a region covered by the first imaging device120a. Similarly, in case where the second person P2is being tracked based upon the object detection technology or algorithm, the first imaging device120amay hand-off all tracking characteristics associated with the second person P2to the fifth imaging device200when the second person P2is about to enter or enters a region covered by the fifth imaging device200. As the case may be, the first imaging device120aor the fifth imaging device200may then transmit the image data to the display screen118based on the associated and received parameters of the fifth imaging device200or the first imaging device120a, respectively.

FIG. 6shows the display screen118in a configuration where the first person P1is being selected and tracked. In this configuration, the first zone Z1includes the first person P1. The first zone Z1also shows an area A selected on the display screen118by a hand600of a user (not shown), that is, a user input600. The selected area A defines a region of interest RI having the person P1. The region of interest RI may include a plurality of region of interests RI spread across same or various zones. In another embodiment, the plurality of region of interests RI includes an associated priority rank (not shown) for each region of interest RI. The associated priority rank may be defined by a size (not shown) of corresponding region of interest RI.

On receiving the user input600at the display screen118, the central imaging device202may send a signal to request the image data generated by the first imaging device120acorresponding to the region of interest RI to the display screen118in an uncompressed form. In an embodiment, the first imaging device120aor the central imaging device202may automatically identify the region of interest RI. In an embodiment where the plurality of region of interests RI are being considered, the associated priority ranks may be used to prioritize the image data before sending to the display screen118. It may be contemplated that except selection by user input and an automatic selection by the central imaging device202, the region of interest RI may be automatically selected by a controller (not shown) based upon the object detection technology or algorithm, used in the art. In an embodiment, the object detection technology or algorithms may include any technique of identifying objects in the captured image data.

FIG. 7depicts a work tool visibility enabled machine100in various images.FIG. 7includes a first image700, a second image702, and a third image704. The first image700depicts a view of the environment E of the machine100captured by the fifth imaging device200. A first area A1is illustrated as selected by the controller for providing additional visibility based on the associated parameter of the fifth imaging device200and the received parameter of the central imaging device202. Similarly, the second image702depicts a view of the environment E of the machine100captured by the first imaging device120a. A second area A2is illustrated as selected by the controller for providing additional visibility based on the associated parameter of the first imaging device120aand the received parameter of the central imaging device202. The third image702depicts a view captured by the central imaging device202.

As illustrated inFIG. 8, the first image700and the second image702are merged with the third image704for providing additional visibility. The first area A1and the second area A2are the additional visible regions in the respective images and are sent to the central imaging device202in a compressed or an uncompressed form for combining with the third image704. The central imaging device202also transmits the merged image with compression of some image data to the display118.

With combined reference toFIG. 1-8, the first imaging device120agenerates the first image data. The first imaging device120astores the generated first image data in the uncompressed form. The first imaging device120asimultaneously compresses the first image data. In an embodiment, the first imaging device120atransmits the compressed or the uncompressed first image data to the central imaging device202.

The first imaging device120afurther generates the first signal indicative of the first image data. The first imaging device120aor the central imaging device202sends the first image data to the display screen118over a data link (not shown) of the machine100. The first person P1is selected and being tracked based upon the object detection technology or algorithm and when the first person P1migrates between multiple zones, the first imaging device120amay hand-off all tracking characteristics associated with the first person P1to the respective imaging device. Similarly, the first area A1and the second area A2are passed on to the central imaging device202in an uncompressed form being the region of interests RI. In some embodiments, the first area A1and the second area A2are passed on to the central imaging device202in a compressed form even after being the region of interest RI because of an exceptional bigger size of the region of interest RI. The uncompressed image data belonging to the first area A1and the second area A2are received and merged by the central imaging device202for providing work tool visibility.

In an embodiment where the plurality of region of interests RI are being considered, the associated priority ranks may be used to prioritize the data before sending to the display screen118. In an embodiment, the first imaging device120a, the second imaging device120b, the third imaging device120c, and the fourth imaging device120dgenerate and communicate the first image data, the second image data, the third image data, and the fourth image data, respectively, to the first zone Z1, the second zone Z2, the third zone Z3, and the fourth zone Z4of the display screen118for generating a surround-view image of the environment E of the machine100.

INDUSTRIAL APPLICABILITY

The present disclosure provides a method of displaying the image data of the environment E of the machine100. A method900for displaying the image data is illustrated with the help ofFIG. 9. In an embodiment, the machine100is switched on and is operating to excavate.

The method900at step902includes generating the image data of the environment E around the machine100by the plurality of imaging devices120having the respective associated operating parameter. In various embodiments, the associated operating parameter may include, for example, the orientation of the plurality of imaging devices120with respect to each other, quality of the images, and the time at which the image data is being generated etc. The method900at step904includes receiving the operating parameter associated with at least one another imaging device120from the plurality of imaging devices120.

The method900at step906includes transmitting the image data to the display screen118over the data link of the machine100by the imaging device120based on the associated and received operating parameters. Selectively sending the image data based on the received and associated operating parameters improves the tracking of any personnel or object around the machine100. Similarly, selectively handing-off the tracking characteristics from one imaging device120to another based on the associated or the received parameters, for example, the orientation of the plurality of imaging devices120relative to each other, the comparative quality of images captured by the plurality of imaging devices120, and the time stamp etc., enables smooth transmission of the image data over the existing data link adhering to current system design requirements. The method900at step908includes displaying the image data at the display screen118over the data link of the machine100by the imaging device120.

The method900may further include receiving the transmitted image data from the plurality of imaging devices120by the central imaging device202. In an embodiment, the method900may include receiving the transmitted image data from the plurality of imaging devices120by the central imaging device202. The method900may further include transmitting the image data corresponding to the region of interest RI in the compressed form to the display screen118. The ability of the central imaging device202to handle a process such as on-board compression enables utilization of a cost-effective option for ECU and also enables retrofitting of the display system300in older machines without replacing the existing ECU with limited processing power and memory. The method900may also include transmitting the uncompressed image data to the display screen118by the central imaging device202.

Additionally, the method900may include defining the associated parameters of the plurality of imaging devices120by the orientation of the imaging device120relative to the plurality of imaging devices120. The associated parameters of the plurality of imaging devices120may be further defined by the image quality and the time at which the image data is being generated etc. The associated parameters of the plurality of imaging devices120may be also defined by both the orientation of each of the plurality of imaging devices120relative to every other, and the image quality. The use of the associated parameters like orientation, image quality, and time etc. enables selective transmission of the image data over the data link improving tracking, object or personnel detection, and work tool visibility, respectively.