METHOD FOR MAINTAINING EQUIPMENT BY REMOTE OPERATION AND ASSISTANCE AND ELECTRONIC DEVICE

A method for maintaining equipment by remote operation and assistance, applied in an electronic device, obtains on-site information in response to a failure notification, and queries maintenance guidance information according to the on-site information. When the maintenance guidance information exists, the electronic device displays or plays the maintenance guidance information by Augmented Reality (AR) glasses, collects user operation data of a maintenance operation in real time, compares the user operation data with the maintenance guidance information, and prompts a user to correct a maintenance operation when a comparison indicates that a current repair or maintenance operation is erroneous. When the maintenance guidance information does not exist, the electronic device establishes a connection with a remote expert terminal by the AR glasses, and obtains real-time guidance from the remote expert terminal.

This application claims priority to Chinese Patent Application No. 202111274278.4 filed on Oct. 29, 2021, the contents of which are incorporated by reference herein.

FIELD

The subject matter herein generally relates to an industrial maintenance field, in particular, relates to a method for maintaining equipment by remote operation and assistance and an electronic device.

BACKGROUND

Maintenance works on different equipment become more complex as the equipment becomes more complex. After a placement machine fails, senior engineers are required to perform maintenance, and perhaps experts and manufacturers are required to provide assistance. Maintenance skills of on-site maintenance personnel are uneven, and maintenance efficiency is low, the equipment utilization rate is affected. Sparse experts, high cost of technical assistance, long training cycle for new recruits, and lack of experience will lead to problems such as low maintenance efficiency, heavy time-consumption and labor-intensive work, with poor safety.

Furthermore, traditional training methods are not efficient, and enterprises often need to spend a lot of training time and expenses. Learners cannot be productive right away, thus consuming the time and energy of teaching staff. At present, although there are a few application softwares in the field of industrial maintenance that can provide auxiliary maintenance functions, the maintenance needs of specific equipment such as placement machines are not met, nor can it meet the maintenance needs of specific faults. In addition, most of these softwares need to be connected to the Internet to run, and for some enterprises with high security and confidentiality requirements, the maintenance site is not connected to an external network.

DETAILED DESCRIPTION

The present disclosure, including the accompanying drawings, is illustrated by way of examples and not by way of limitation. Several definitions that apply throughout this disclosure will now be presented. It should be noted that references to “an” or “one” embodiment in this disclosure are not necessarily to the same embodiment, and such references mean “at least one”.

FIG.1illustrates an application architecture of a method for maintaining equipment by remote operation and assistance. The method is applied in a terminal device101, a cloud server104, and a remote expert terminal107. The terminal device101is used by the user to collect on-site information. The terminal device101at least includes Augmented Reality (AR) glasses102. The terminal device101collects the on-site information by using the AR glasses102. The terminal device101transmits the on-site information to the cloud server104by a first 5G intranet103. The cloud server104stores 3D model data, maintenance guidance information, expert cases, historical maintenance records, and on-site information of the device. The backend105is used to manage the maintenance guidance information, the expert cases and the on-site information. The remote expert terminal107obtains the on-site information from the cloud server104by a second 5G intranet106.

FIG.2illustrates the method for maintaining equipment by remote operation and assistance. The method is provided by way of example, as there are a variety of ways to carry out the method. Each block shown inFIG.2represents one or more processes, methods, or subroutines carried out in the example method. Furthermore, the illustrated order of blocks is by example only and the order of the blocks can be changed. Additional blocks may be added or fewer blocks may be utilized, without departing from this disclosure. The example method can begin at block201.

At block201, in response to a failure notification from an electronic device, obtaining on-site information, and querying corresponding maintenance guidance information according to the on-site information.

In one embodiment, the method is applied to electronic equipment, and the electronic equipment includes, but is not limited to, a mounter machine. When the electronic equipment fails, for example, the mounter machine cannot apply a software patch or an update as a solution, or the placement machine cannot accurately patch components to corresponding position, defective products are produced. When personnel on a production line find the above fault in the mounter machine, a failure notification is issued to notify the maintenance personnel to go to repair the faulty equipment.

In one embodiment, when the electronic device fails, the electronic device generates a failure notification. Exemplarily, when the electronic device cannot be patched correctly, a patch error fault prompt will be displayed on a display screen of the electronic device.

FIG.3illustrates collecting on-site information. In one embodiment, obtaining the on-site information includes: collecting on-site maintenance video of a faulty equipment by the high-definition surveillance camera301inside the faulty equipment, and collecting real-time video or picture of the faulty equipment by a high-definition camera302on the AR glasses102. The high-definition surveillance camera301collects maintenance videos of users, and is used to determine surrounding environment video when the faulty equipment is being repaired. The high-definition camera302on the AR glasses102collects the real-time video or the pictures of the faulty equipment, which helps the user to further determine a location of a fault on the faulty equipment, or to provide real-time maintenance guidance and suggestions for the remote expert terminal107.

In one embodiment, the backend105includes a maintenance guidance sub-module401.FIG.4illustrates the maintenance guidance sub-module. The maintenance guidance sub-module401is used to store and manage maintenance guidance information and expert cases. In one embodiment, the maintenance guidance information includes an operation video and operation instructions for completing a maintenance process for the fault of the faulty equipment. In one embodiment, the maintenance process and expert cases are displayed by a 3D model, so that the user can quickly understand how to perform maintenance. The 3D model data is stored in the cloud server104.

In one embodiment, the remote expert terminal107manages the maintenance guidance information by the backend105of the cloud server104. In one embodiment, the remote expert terminal107supplements or adds the original maintenance guidance information.

FIG.5illustrates an interaction between terminal equipment and remote expert terminal, and cloud server. In one embodiment, the terminal device101scans a two-dimensional code of the faulty equipment by the AR glasses102and communicates with the cloud server104by the interaction module501, and obtains historical maintenance information of the faulty equipment from the cloud server104. In one embodiment, the terminal device101transmits the on-site information to the cloud server104, and retrieves corresponding maintenance guidance information from the cloud server104. In one embodiment, the terminal device101communicates with the remote expert terminal107by an intranet communication module502. The intranet communication module502includes the first 5G intranet103and the second 5G intranet106.

In one embodiment, the terminal device101scans the two-dimensional code of the faulty equipment by the AR glasses102, obtains the historical maintenance information of the faulty equipment from the cloud server104, and determines the maintenance guidance information from the historical maintenance information according to selection by user. In one embodiment, the historical maintenance information includes maintenance records of the faulty equipment and maintenance instruction information of the maintenance records. When the historical maintenance information of the faulty equipment is acquired, user determines keywords according to the location where the electronic device has failed, and inputs the keywords into a search box of the AR glasses102to search for the corresponding historical maintenance information. When the corresponding historical maintenance information cannot be found, the terminal device101returns to the search box and continues to search by the search box. For example, for an operation failure of the mounter machine, the user determines keywords such as “power failure”, and enters the keywords into the search box of the AR glasses102to make a search. When finding the corresponding historical maintenance information, the terminal device101retrieves the maintenance guidance information corresponding to the historical maintenance information, and assists the user in repairing the faulty equipment.

In one embodiment, the terminal device101scans the two-dimensional code of the faulty equipment by the AR glasses102, obtains the historical maintenance information of the faulty equipment from the cloud server104, and matches the on-site information with the historical maintenance information, and determines that the historical maintenance information matching the on-site information is the maintenance instruction information. When the historical maintenance information of the faulty equipment is obtained, the terminal device101matches the real-time video or picture of the faulty equipment collected by the AR glasses102with the historical maintenance information picture or video of the AR glasses102, and determines the historical maintenance information matching with the real-time video or picture of the faulty equipment, and the real-time video or picture of the faulty equipment is collected by the AR glasses102. For example, by parsing the real-time videos of the faulty equipment collected by the AR glasses102, the terminal device101intercepts the real-time videos to obtain key video parts, inputs the key video parts into the matching box, and obtains the historical maintenance information matching with the real-time video of the faulty equipment, and takes the historical maintenance information matching with the real-time video of the faulty equipment as the maintenance guidance information.

At block202, determining whether the maintenance guidance information is existed.

In one embodiment, when the keyword is entered into the search box, if the corresponding historical maintenance information can be obtained, it is determined that the maintenance guidance information exists. If the corresponding maintenance guidance information is not obtained, it is determined that the maintenance guidance information does not exist.

In one embodiment, if the historical maintenance information of the faulty equipment contains historical maintenance information that matches the real-time video of the faulty equipment collected by the AR glasses102, it is determined that maintenance guidance information exists. If the historical maintenance information of the faulty equipment does not match the faulty equipment collected by the AR glasses102, the real-time video not matching the historical maintenance information will determine that the maintenance guidance information does not exist.

At block203, when the maintenance guidance information exists, displaying or playing the maintenance guidance information by the AR glasses102.

In one embodiment, when the maintenance guidance information is existing, the terminal device101calls up the maintenance guidance information from the cloud server104by the AR glasses102, displays or plays the content of the maintenance guidance information, and enables repair of the faulty equipment. In one embodiment, the maintenance guidance information is displayed on the AR glasses102, so that dedicated/experienced users can be tasked to repair the faulty equipment and improve maintenance efficiency.

In one embodiment, when the user repairs the faulty equipment according to the maintenance guidance information, the AR glasses102synchronously collects and records user operation data of an maintenance operation in real time, tracks user operation data and compares the user operation data with the maintenance guidance information. For example, by comparing multiple images of point of fault of the faulty equipment within a preset time, or multiple frames of images captured from the video and images of the parts, with the parts of the fault point included in the maintenance guidance information in a non-fault state of the faulty equipment, this determines whether the operation data is abnormal. If the tracking data compared to the maintenance guidance information is abnormal, a warning message will be output to remind the user that there is an error in the maintenance operation, or in a maintenance procedure or a maintenance method, and that the error should be corrected in time. For example, the electronic equipment cannot attract the iron sheet, resulting in the product not being affixed with the corresponding iron sheet. The user may obtain the solution according to the maintenance guidance information as being that a nozzle of the electronic equipment needs to be replaced. When the user replaces the nozzle according to the maintenance guidance information and the replaced nozzle is inconsistent with the nozzle model properly used by the electronic equipment, the warning message will be output to remind the user that there is an error in the maintenance operation, and check whether the maintenance operation is correct until the user replaces the nozzle with a nozzle properly used in the electronic equipment, than the maintenance procedure is completed.

At block204, when the maintenance guidance information does not exist, establishing a communication connection with the remote expert terminal107by the AR glasses102, and obtaining real-time maintenance guidance suggestions from the remote expert terminal107.

In one embodiment, when the maintenance guidance information does not exist, the user cannot carry out maintenance of the faulty equipment, the terminal device101establishes a communication connection with the remote expert terminal107by the AR glasses102. Further, the AR glasses102transmits the collected on-site information by the first 5G intranet103to the cloud server104to save the on-site information and the remote expert terminal107can obtain the on-site information from the cloud server104. After the terminal device101establishes a communication connection with the remote expert terminal107, the remote expert terminal107calls the on-site information from the cloud server104by the second 5G intranet106, communicates with the user according to the on-site information, and gives maintenance guidance suggestions.

In one embodiment, the AR glasses102establish a network connection with the remote expert terminal107by the 5G intranet. The user of the AR glasses102initiates a video call with a remote expert of the remote expert terminal107by the AR glasses102. When the remote expert receives the video call, a video conference call room is established. In the video conference call room, the user or the remote expert can invite other users or other remote experts to join. When the user who originally initiated the video call chooses to end the video call, the video conference call room will be dismissed, and all the invited remote experts or other users will exit the video conference call room.

In one embodiment, all 5G intranets use adaptive transmission mode to transmit the on-site information. The adaptive transmission mode can dynamically adjust the definition of the video in the on-site information according to the bandwidth of the 5G intranet, preventing the remote expert terminal107from being stuck or hanging when viewing the real-time video or picture of the faulty equipment.

In one embodiment, the call records generated after the AR glasses102are used to establish the communication connection with the remote expert terminal107are saved to the cloud server104and managed by the backend105of the cloud server104, including: saving valuable call records and the on-site information, saving valuable call records and on-site information, and deleting worthless call records and on-site information. In one embodiment, the valuable call records or the on-site information include, but are not limited to, significant reference and information on parts of the electronic equipment that frequently fails. Worthless call records or on-site information include, but are not limited to, no significant reference being found and information which is irrelevant.

The above are only specific embodiments of the present application, but are not to limit the protection scope of the present application. For those of ordinary skill in the art, improvements can be made without departing from the inventive concept of the present application, but these all belong to the protection scope of the present application.

FIG.6illustrates the electronic device. The electronic device6includes a storage601, at least one processor602, and a computer program603stored in the storage601and executed by the processor602, and at least one communication bus604.

Those skilled in the art can understand that the schematic diagram shown inFIG.6is only an example of the electronic device6, and does not constitute a limitation on the electronic device6. Other examples may include more or less components than those shown in the drawings, or have different combinations thereof, or different components, for example, the electronic device6may also include input and output devices, network access devices, and the like.

The at least one processor602may be a Central Processing Unit (CPU), and may also be a general-purpose processor, a Digital Signal Processors (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The at least one processor602can be a microprocessor or the at least one processor602can also be any conventional processor, etc. The at least one processor602is the control center of the electronic device6, using various interfaces and lines to connect various parts of the entire electronic device6.

The storage601can be used to store the computer program603, and the at least one processor602implements the electronic program by executing the computer program603stored in the storage601and calling up the data stored in the storage601. The storage601may mainly include a stored program area and a stored data area, wherein the stored program area may store an operating system, and at least one application program required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The storage data area may store data (such as audio data) created in the use of the electronic device6, etc. In addition, the storage601may include non-volatile storage such as a hard disk, an internal memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card (Flash Card), at least one disk storage device, flash memory device, or other non-volatile solid state storage device.