Abstract:
An agricultural vehicle including a chassis, a controller carried by the chassis, and a vehicle maintenance system coupled to the vehicle. The vehicle maintenance system includes a viewing system configured to process an image of at least a portion of the vehicle and an augmentation system. The augmentation system is configured to identify an element of the vehicle in the image; receive information from the vehicle about a maintenance action relative to the element; and place a designator proximate to the element identified in the image, the designator indicating the maintenance action that is to be undertaken involving the element.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    1. Field of the Invention 
         [0002]    The present invention relates to vehicles, and, more particularly, to an apparatus to direct the maintenance of an agricultural vehicle. 
         [0003]    2. Description of the Related Art 
         [0004]    Vehicles in general and agricultural vehicles in particular require maintenance and/or adjustments of their subsystems. An agricultural combine, for example, is used to harvest agricultural crops such as corn, soybeans, wheat and other grain crops. As the combine is operated maintenance items arise, either as a passage of operation or as detected by sensors on the combine. 
         [0005]    Augmented reality (AR) systems provide a live direct or indirect view of a physical, real-world environment whose elements are augmented or supplemented by way of a computer-generated sensory input such as sound, video, graphics or GPS data. AR systems are used to enhance a person&#39;s current perception of reality. The augmentation process is conventionally carried out in real-time and in context with detected or known environmental elements, such as a battlefield environment. The AR technology includes object recognition allowing the information about the viewed environment to become interactive. Information about the environment and its objects can be overlaid on the view of the real world. 
         [0006]    What is needed in the art is a vehicular maintenance system that uses AR and allows for easy identification of the item to be maintained and with information on the way to perform the maintenance. 
       SUMMARY OF THE INVENTION 
       [0007]    The present invention provides an agricultural vehicle with an onboard maintenance record system that interacts with an AR system to assist the operator with maintenance actions. 
         [0008]    The invention in one form is directed to an agricultural vehicle including a chassis, a controller carried by the chassis, and a vehicle maintenance system coupled to the vehicle. The vehicle maintenance system includes a viewing system configured to process an image of at least a portion of the vehicle and an augmentation system. The augmentation system is configured to identify an element of the vehicle in the image; receive information from the vehicle about a maintenance action relative to the element; and place a designator proximate to the element identified in the image, the designator indicating the maintenance action that is to be undertaken involving the element. 
         [0009]    The invention in another form is directed to a vehicle maintenance system that includes a viewing system configured to process an image of at least a portion of the vehicle and an augmentation system. The augmentation system is configured to identify an element of the vehicle in the image; receive information from the vehicle about a maintenance action relative to the element; and place a designator proximate to the element identified in the image, the designator indicating the maintenance action that is to be undertaken involving the element. 
         [0010]    The invention in yet another form is directed to a vehicle maintenance method including the steps of: viewing a portion of the vehicle with a viewing system, the viewing system displaying an image; identifying an element of the vehicle in the image; receiving maintenance information about the element from the vehicle; and augmenting the image by placing a designator proximate to the element identified in the image, the designator including information received in the receiving step. 
         [0011]    An advantage of the present invention is that all of the maintenance information for the vehicle is contained in the vehicle. 
         [0012]    Another advantage of the present invention is that the maintenance system uses interactive helps and prompting of the user by way of a device that a lot of people are already familiar with, such as a smart phone. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0013]    The above-mentioned and other features and advantages of this invention, and the manner of attaining them, will become more apparent and the invention will be better understood by reference to the following description of embodiments of the invention taken in conjunction with the accompanying drawings, wherein: 
           [0014]      FIG. 1  is a view of an embodiment of a vehicle maintenance apparatus according to the present invention; 
           [0015]      FIG. 2  is a view of another embodiment of the vehicle maintenance apparatus according to the present invention; 
           [0016]      FIG. 3  is a view of the invention being carried out on a smart phone; and 
           [0017]      FIG. 4  is a block diagram illustrating an interface of a display device coupled with a control system of a vehicle. 
       
    
    
       [0018]    Corresponding reference characters indicate corresponding parts throughout the several views. The exemplifications set out herein illustrate embodiments of the invention and such exemplifications are not to be construed as limiting the scope of the invention in any manner. 
       DETAILED DESCRIPTION OF THE INVENTION 
       [0019]    Referring now to the drawings and more specifically to  FIGS. 1-4  there is shown an agricultural vehicle  10  having a chassis  12 . A viewing system  16  works in cooperation with an augmentation system  18  to display information regarding vehicle  10  on a display  24  thereby providing information and instructions to an operator about vehicle  10  A primary display  22  located in the cab of vehicle  10  alerts the operator that an active fault has been detected or that maintenance is needed. A positioning system  20  assists a controller  26  with determining what designators to place in the viewing area of secondary display  24 . 
         [0020]    The secondary display  24  may be a set of viewing glasses, as shown in  FIG. 1 , a tablet as shown in  FIG. 2 , or a phone as shown in  FIG. 3 , each using a viewing system  16  and the augmentation system  18 . The viewing system  16  can be optic as in the case of the viewing glasses or a produced image as in the case with the other devices. Augmentation system  18  detects the portion of the vehicle  10  being viewed so that information relevant to what is being viewed is shown. 
         [0021]    In  FIG. 1  there is shown a person wearing a display  24 , which is remotely connected to controller  26 . The view of vehicle  10  by the operator is augmented by augmentation system  18  placing designators  28  and  30  there so that information about issues needing attention are brought to the attention of the operator. Here designator  28 , which may be at least partially color coded red to indicate that there is an active fault that needs immediate attention, in that the air filter has a blockage that needs to be cleared, by indicating that service is required. The operator can select which issue is to be currently addressed, or as the operator approaches a portion of the vehicle  10 , such as the air filter, controller  26  understands that this particular issue is to be addressed. 
         [0022]    When the focus is on one issue represented by a designator, then other designators, such as designator  30 , may be temporarily removed from the display, so that a focus on, as in this case, the air filter can be undertaken. More detailed information about the air filter can be shown as the operator approaches the air filter, such as suggested courses of action, what tools are needed, what steps are needed to access the air filter, when the air filter was last serviced or replaced, and when the next regular service is scheduled. 
         [0023]    In  FIG. 2 , another embodiment of the secondary display  24  is illustrated with designator  30  showing that a tie rod needs to be greased. The camera in display  24  is directed at vehicle  10  and the image is processed so that elements of vehicle  10  that appear in the image are identified so that designator  30  is directed to the right element. Controller  26 , which is located in vehicle  10 , although other locations are also contemplated, keeps track of the maintenance requirements and operational faults of vehicle  10  so that it can be properly maintained. Here, controller  26  has determined that the tie rod is due to be greased, hence designator  30  is positioned with an arrow generally pointing to where the tie rod is located. 
         [0024]    In  FIG. 3 , another embodiment of the secondary display  24  is illustrated in the form of a phone that is wirelessly coupled to controller  26  that is illustrating a split screen, with the left side having the image of vehicle  10  with a designator  32  indicating that the rotor variator has an upcoming maintenance action that is due in 3 hours. On the right half of the screen a detailed image is sent by controller  26  to illustrate to the operator what is to be looked for as the operator approaches a part of vehicle  10 . Again the messages may be changed as the position of the operator changes, as detected by positioning system  20 , so that more pertinent information is conveyed to the operator. Yet further, if the operator needs to have a tutorial to illustrate what is to be done the operator can select the video assistance button  36  and controller  26  sends a video of the actions needed to address the maintenance issue. Once the operator has completed the operation controller  26  notes the completion in the record and schedules the next maintenance event. It is also contemplated that display  24  may record a portion of the maintenance or even request that the operator scan the area with display  24  so that a visual record of the maintenance is preserved by controller  26  in the maintenance log of vehicle  10 . 
         [0025]    The present invention uses augmented reality (AR) with a tablet, smart phone, or set of computerized glasses, each being referred to as secondary display  24 , communicating with a machine  10  over a wireless network to project a floating indicator  28 ,  30 ,  32  pointing out all inspection and service points (wear parts, lube points, etc.) on the machine (as seen on the screen of the tablet by use of the camera), combined with the engine hours of the machine  10  to indicate (for example, during a vehicle walk around) the number of hours left on the service point (service point arrow and indicator turns yellow if less than 20 hours and greater than 8 hours, red if less than 8 hours). When the user selects the service point on the machine  10 , the display  24  will go to split screen mode (as shown in  FIG. 3 ), and the service manual  34  and/or operator manual pages relevant to the selected service point will be displayed. After successful completion of the service, the counter is reset for that particular service item. Historical service records are saved in the system memory to create a report showing all completed services over the life of the vehicle, which serves to increase the value of the used vehicle. 
         [0026]    The Augmented Reality App  18  receives information wirelessly from the machine  10  or mechanism  10  for superimposing information thereon. For example, you point your camera on your device directly at the machine  10 . On your display you see a red flag that&#39;s pointing directly at the device that needs attention, in the body of the red flag, there is shown a data field that is populated by way of the wireless communication from the machine  10 . For example, the data may be 3 hours left before the service interval is expired. After the service is completed, the user would touch the red flag on the screen, then a box would appear “service complete”, then the device  24  communicates back to the machine  10  that the service is complete, and that interval would be reset for attention in 500 hours, for example. 
         [0027]    As another example of the Augmented Reality App receiving information wirelessly from the machine, there could be an active fault on the machine  10 , such as shown in designator  28 , with an air filter being partially blocked. When the user points the camera of his device  24  at the machine  10  engine bay area, on the display  24  you would see a red flag that is pointing directly at the air filter. In the body of the red flag, it shows a data field that is populated via wireless communication from the machine  10 , which could be, for example, “Filter Partially Plugged, Filter Service Required”. 
         [0028]    While this invention has been described with respect to at least one embodiment, the present invention can be further modified within the spirit and scope of this disclosure. This application is therefore intended to cover any variations, uses, or adaptations of the invention using its general principles. Further, this application is intended to cover such departures from the present disclosure as come within known or customary practice in the art to which this invention pertains and which fall within the limits of the appended claims.