Brevet FR3133260A1 — publié le 20230908

Disclosed are a method and apparatus for converting 3D manuals into 2D interactive videos for a cloud service. The method for converting 3D manuals into 2D interactive videos for a cloud service, proposed in the present disclosure, includes the steps of performing labeling for annotations in a 3D manual for a cloud service and machine learning for recognizing the annotations; automatically recognizing the annotations, fine-adjusting automatically unrecognized annotations, and generating a file for converting the 3D manual into a 2D interactive video; linking the 3D manual to a corresponding 2D manual using the file; extracting and handling artificial intelligence knowledge based on work instructions in the 2D manual; and performing neural network tasks and linguistic inferences for processing voice requests, and informing the user about the handling and completion of a requested task by sending visual and audio feedback, in order to execute commands based on the artificial intelligence knowledge, the digital twin, and the annotations in the 3D manual. of the invention: Method and apparatus for converting 3D manuals into 2D interactive videos for cloud service Cross-references to related application This application is based on and claims priority to Korean Patent Application No. 10-2022-0026672, filed on 02.03.2022 in the Korean intellectual property office, the disclosures of which are herein incorporated by reference in its entirety. The present disclosure relates to a method and apparatus for converting 3D manuals into 2D interactive videos for a cloud service. Background of the invention For complicated technical manuals such as on aircraft, a highly qualified expert may create annotations by means of a video annotation tool dedicated for a video to be annotated. In other cases, even a non-expert may create simplified annotations simply by placing text on the screen. In these cases, the created annotations are mostly static annotations that are displayed in a fixed position. Further, these annotations are provided to a viewer, unilaterally or in the form of static text or image without being moved. In some cases, the annotations may interact with the user, but only at a level that provides a link to another video or web site. For example, Korean Registered Patent No. 10-1072347 issued on 5 October, 2011 discloses a technology that creates situation information on a given scene by using text information. As described above, even if the user is unilaterally provided with a written text or image annotation for a video in a unilateral manner, they are not able to interact with the video and get information from the annotation or add information to it. Although the user may have the need for acquisition of various information and interaction through annotations, these simplified annotations are merely shown unilaterally and statically, or interactions are provided simply in the form of a link. Thus, the user often has to do some extra work off the screen in order to acquire various information and perform an additional task. For example, when the user performs work as instructed in a complicated technical manual such as on aircraft, they usually refer to instructions for assembly and disassembly and to paper-based 2D engineering drawings for comparison with real-world three-dimensional mechanical parts. Since they are paper-based 2D manuals, it may be hard to match the part numbers with the parts of the real-world physical machine, and it takes some time to understand how they match, which is time-consuming and less efficient and therefore often cause errors. [Prior Art Document] Korean Registered Patent Publication No. 10-1072347 (issued on 5 October, 2011) Summary A technical aspect of the present disclosure is to provide a method and apparatus for converting 3D manuals into 2D interactive videos for a cloud service. More specifically, the present disclosure provides a method and apparatus for creating a three-dimensional digital twin of a real machine through XR (eXtended Reality) technology, for a complicated 3D manual such as on aircraft, and for additionally providing instructions, animations, and 2D engineering drawings on the screen which are required for assembly and disassembly. In one aspect, a method for converting 3D manuals into 2D interactive videos for a cloud service, proposed in the present disclosure, includes the steps of performing labeling for annotations in a 3D manual for a cloud service and machine learning for recognizing the annotations; automatically recognizing the annotations, fine-adjusting automatically unrecognized annotations, and generating a file for converting the 3D manual into a 2D interactive video; linking the 3D manual to a corresponding 2D manual using the file; extracting and handling artificial intelligence knowledge based on work instructions in the 2D manual; and performing neural network tasks and linguistic inferences for processing voice requests, and informing the user about the handling and completion of a requested task by sending visual and audio feedback, in order to execute commands based on the artificial intelligence knowledge, the digital twin, and the annotations in the 3D manual. In the step of performing labeling for annotations in a 3D manual for a cloud service and machine learning for recognizing the annotations by means of a learning part, the positions of annotation objects in the 3D manual are represented in a coordinate system, and labeling for annotations in the 3D manual is performed so as to automatically recognize the positions of the annotations, and, after the labeling, machine learning is performed to automatically recognize annotations on objects in the video through an object detection model. In the step of automatically recognizing the annotations, fine-adjusting automatically unrecognized annotations by means of a video editor, and generating a file for converting the 3D manual into a 2D interactive video, the automatically unrecognized annotations are adjusted by adjusting the positions and shapes of the annotations and then adjusting the durations of the annotations by means of the video editor, and the automatically recognized annotations represent a list of taks for interaction, including object information, parts information, marks for proceeding to a next task or returning to a previous task, parts positions, relevant videos, and caution and warning buttons. In the step of linking the 3D manual to a corresponding 2D manual using the file by means of a 2D manual linking part, the 3D manual is linked to the corresponding 2D manual using the file which includes the positions of 2D drawings in a video, object label names, the start and end times of bounding boxes for interaction, and the positions of the bounding boxes in a video frame, and a corresponding interaction is performed when an annotation in the video is clicked. In another aspect, an apparatus for converting 3D manuals into 2D interactive videos for a cloud service, proposed in the present disclosure, includes: performing labeling for annotations in a 3D manual for a cloud service and machine learning for recognizing the annotations; automatically recognizing the annotations, fine-adjusting automatically unrecognized annotations, and generates a file for converting the 3D manual into a 2D interactive video; linking the 3D manual to a corresponding 2D manual using the file; extracting and handling artificial intelligence knowledge based on work instructions in the 2D manual; and performing neural network tasks and linguistic inferences for processing voice requests, and informing the user about the handling and completion of a requested task by sending visual and audio feedback, in order to execute commands based on the artificial intelligence knowledge, the digital twin, and the annotations in the 3D manual. A method and apparatus for converting 3D manuals into 2D interactive videos for a cloud service according to embodiments of the present disclosure may create a three-dimensional digital twin of a real machine through XR (eXtended Reality) technology, for a complicated 3D manual such as on aircraft, and additionally provide instructions, animations, and 2D engineering drawings on the screen which are required for assembly and disassembly. Description of the drawings The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein: is a view showing a 3D manual according to an embodiment of the present disclosure and a 2D manual according to the conventional art. is a view showing a construction of an apparatus for converting 3D manuals into 2D interactive videos for a cloud service according to an embodiment of the present disclosure. is a flowchart for explaining a method for converting 3D manuals into 2D interactive videos for a cloud service according to an embodiment of the present disclosure. is a view for explaining a process of representing the positions of annotation objects according to an embodiment of the present disclosure. and; is a view for explaining a process of performing labeling for annotations according to an embodiment of the present disclosure. is a view for explaining a machine learning process according to an embodiment of the present disclosure. is a view showing an example of use of an automatic object detection model according to an embodiment of the present disclosure. is a view for explaining a process of linking a 3D manual to a corresponding 2D manual according to an embodiment of the present disclosure. is a view showing a structure of 3D manual assets for interaction according to an embodiment of the present disclosure. and; is a view showing an automatically structured file according to an embodiment of the present disclosure. A video conversion method comprising the steps of: performing labeling for annotations in a 3D manual for a cloud service and machine learning for recognizing the annotations; automatically recognizing the annotations, fine-adjusting automatically unrecognized annotations, and generating a file for converting the 3D manual into a 2D interactive video; linking the 3D manual to a corresponding 2D manual using the file; extracting and handling artificial intelligence knowledge based on work instructions in the 2D manual; and performing neural network tasks and linguistic inferences for processing voice requests and informing the user about the handling and completion of a requested task by sending visual and audio feedback, in order to execute commands based on the artificial intelligence knowledge, the digital twin, and the annotations in the 3D manual. The video conversion method of claim 1, wherein, in the step of performing labeling for annotations in a 3D manual for a cloud service and machine learning for recognizing the annotations by means of a learning part, the positions of annotation objects in the 3D manual are represented in a coordinate system, and labeling for annotations in the 3D manual is performed so as to automatically recognize the positions of the annotations, and, after the labeling, machine learning is performed to automatically recognize annotations on objects in the video through an object detection model. The video conversion method of claim 1, wherein, in the step of automatically recognizing the annotations, fine-adjusting automatically unrecognized annotations by means of a video editor, and generating a file for converting the 3D manual into a 2D interactive video, the automatically unrecognized annotations are adjusted by adjusting the positions and shapes of the annotations and then adjusting the durations of the annotations by means of the video editor, and the automatically recognized annotations represent a list of taks for interaction, including object information, parts information, marks for proceeding to a next task or returning to a previous task, parts positions, relevant videos, and caution and warning buttons. The video conversion method of claim 1, wherein, in the step of linking the 3D manual to a corresponding 2D manual using the file by means of a 2D manual linking part, the 3D manual is linked to the corresponding 2D manual using the file which includes the positions of 2D drawings in a video, object label names, the start and end times of bounding boxes for interaction, and the positions of the bounding boxes in a video frame, and a corresponding interaction is performed when an annotation in the video is clicked. A video conversion apparatus comprising: a processor; a memory coupled to the processor, the memory containing instructions, that when executed by the processor, perform the steps of: performing labeling for annotations in a 3D manual for a cloud service and machine learning for recognizing the annotations; automatically recognizing the annotations, fine-adjusting automatically unrecognized annotations, and generating a file for converting the 3D manual into a 2D interactive video; linking the 3D manual to a corresponding 2D manual using the file; extracting and handling artificial intelligence knowledge based on work instructions in the 2D manual; and performing neural network tasks and linguistic inferences for processing voice requests and informing the user about the handling and completion of a requested task by sending visual and audio feedback, in order to execute commands based on the artificial intelligence knowledge, the digital twin, and the annotations in the 3D manual. The video conversion apparatus of claim 5, wherein the labeling represents the positions of annotation objects in the 3D manual in a coordinate system and performs labeling for annotations in the 3D manual so as to automatically recognize the positions of the annotations, and, after the labeling, performs machine learning to automatically recognize annotations on objects in the video through an object detection model. The video conversion apparatus of claim 5, wherein the automatically recognizing adjusts the automatically unrecognized annotations by adjusting the positions and shapes of the annotations and then adjusting the durations of the annotations, and the automatically recognized annotations represent a list of taks for interaction, including object information, parts information, marks for proceeding to a next task or returning to a previous task, parts positions, relevant videos, and caution and warning buttons. The video conversion apparatus of claim 5, wherein the linking links the 3D manual to the corresponding 2D manual using the file which includes the positions of 2D drawings in a video, object label names, the start and end times of bounding boxes for interaction, and the positions of the bounding boxes in a video frame and performs a corresponding interaction when an annotation in the video is clicked.