Patent Publication Number: US-2021185280-A1

Title: Method and system for teletransmitting to a processing site a video stream captured on a remote intervention site

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a national phase entry under 35 U.S.C. § 371 of International Patent Application PCT/FR2019/051070, filed May 13, 2019, designating the United States of America and published as International Patent Publication WO 2019/220043 A1 on Nov. 21, 2019, which claims the benefit under Article 8 of the Patent Cooperation Treaty to French Patent Application Serial No. 1854006, filed May 14, 2018. 
    
    
     TECHNICAL FIELD 
     The present disclosure concerns a method for teletransmitting to a processing site a video stream captured on a remote intervention site. It also concerns a system implementing the teletransmission method. 
     BACKGROUND 
     One particular field of the present disclosure is teleassistance for operations of all kinds performed on remote intervention sites, particularly industrial operations such as preventive or corrective maintenance operations or inspection operations. 
     Another field of the present disclosure is the captured video stream teletransmission from mobile, terrestrial, aerial or aquatic platforms, particularly for inspection purposes. 
     The teletransmission of information from a remote intervention site is currently performed by using numerous communication vectors, whether wired via communication networks or wireless using wireless or satellite communication. 
     In the field of teleassistance, the teletransmitted information generally comprises video streams generated from video capture equipment employed by a field operator. 
     In video teleassistance, field operators load a camera intended to capture a stream of images of equipment on which maintenance or operations will be performed. Teleassistance can either be synchronous or asynchronous. The video stream is transmitted by digital communication means, such as the IP network via satellite or a land-based network. 
     The tele-expert receives the stream, analyzes it, and returns information to the operator to assist him. This information can include multimedia data and/or procedural instructions. 
     Document EP 2765502 A1 thus discloses a teleassistance method contributing to providing enhanced information to a field operator. 
     Document WO 2009128781 also discloses a remote display method and device. 
     Document US 2017/300757 A1 discloses an image capture method and system providing protection for confidentiality and privacy. It does not describe remote assistance of a field operator on an intervention scene, no stream of information being provided from the processing site to the intervention scene. 
     One disadvantage of the current methods is found in the excessive comprehensiveness of the transmitted information. The captured and transmitted information can contain superfluous information, some of it even secret, which can present risks in terms of security of facilities or of economic or technological intelligence. 
     The purpose of the present disclosure is to propose a teletransmission method intended to reduce the transmission of information solely to data that are necessary and sufficient to properly carry out an intervention or an inspection. Another purpose of the present disclosure is to optimize the quantity of data transmitted during the teletransmission operation and thus to be less sensitive to the technological limitations of the communication networks used. 
     BRIEF SUMMARY 
     This objective is achieved with a method for teletransmitting to a processing site a video stream captured at a remote scene of intervention, comprising a processing of the video stream in such a way as to erase specific zones, then a transmission of the stream thus processed to the processing site, wherein it is implemented to remotely assist a field operator working on a unit or object having a known geometric shape in the remote scene of intervention. 
     Thus, it becomes possible to avoid having sensitive information contained in the scene of intervention or on the unit itself be intercepted on the communication channels used between an intervention site and an appraisal site. For example, this sensitive information can involve a geographic location, a building or visually identifiable equipment. It can also be information contained on a label or any other medium affixed to the unit undergoing an intervention. Such information can also be of a contextual nature, such as natural or artificial lighting conditions or atmospheric or weather conditions. 
     By using the teletransmission method according to the present disclosure, a teleassistance tool can be proposed on sensitive intervention sites that considerably limits the risk of interception of confidential data by third parties during the teletransmission of information, particularly of images or video stream transmitted by an operator working on site. 
     In one particular implementation of the present disclosure, the teletransmission method further comprises the steps of:
         capturing a video stream relating to an intervention scene including the unit or object,   extracting a limited stream from the unit or object,   processing the extracted video stream in order to erase specific zones,   transmitting the video stream thus processed to a teleassistance site,   interpreting the stream at the teleassistance site, with a view to enriching the stream and/or generating an instruction,   transmitting the enriched stream and/or the instruction from the teleassistance site to the field operator, and   restoring to the field operator the stream and/or the instructions via suitable interfaces.       

     A zone of interest not to be transmitted can be previously known or identified at the scene of intervention. The identification of this zone of interest not to be transmitted can also be performed by a field operator via a suitable graphic interface, or it can be done automatically. 
     In one particular form of the present disclosure, the step of processing the extracted unit further comprises a step for modifying characteristics of the content of the video stream that are extrinsic to the identified object or unit and/or that carry semantic information inherent to the environment of the object or unit. 
     In another embodiment of the teletransmission method according to the present disclosure, the video stream is captured from a platform that is mobile relative to the scene of intervention. This mobile platform can be land-based, aerial or aquatic and can for example be carried by a drone. It can be controlled locally by an operator working on a remote site. 
     According to another aspect of the present disclosure, a system is proposed for teletransmitting to a processing site a video stream captured at a remote scene of intervention by video stream capture means, implementing the teletransmission method according to the present disclosure, comprising, at the scene of intervention, means for processing the video stream so as to erase specific zones and means for transmitting the stream thus processed to the processing site, wherein the method implements, in order to remotely assist from a teleassistance site, a field operator working on a local site on a unit or object having a known or geometric shape. 
     The teletransmission system according to the present disclosure can further comprise:
         means for identifying the unit or object within the video stream, and   means for extracting a limited video stream from the specific identified unit or object.       

     Furthermore, graphic interface means can advantageously be provided on the scene of intervention to enable a field operator to identify the at least one zone of interest not to be transmitted. 
     In one particular version of the present disclosure, the means of processing the extracted unit or object can comprise means for modifying characteristics of the content of the video stream that are extrinsic to the identified unit or object and/or that carry semantic information inherent to the environment of the object or unit. 
     In one particular embodiment of the present disclosure, the teletransmission system can thus comprise:
         on the local site:
           means for capturing video stream relating to an intervention scene including the unit or object,   means for processing the video stream in order to erase specific zones therefrom,   means for transmitting the video stream thus processed to the teleassistance site,   
           on the teleassistance site:
           means for interpreting the stream, with a view to enriching the stream and/or generating an instruction,   means for transmitting the enriched stream and/or the instruction from the teleassistance site to the field operator, and   means for restoring to the field operator the stream and/or the instructions via suitable interfaces.   
               

     In another particular version of a teletransmission system according to the present disclosure, the means for capturing, processing and transmitting the video stream are placed on board a platform that is mobile relative to the scene of intervention. 
     The mobile platform can advantageously be carried by a drone. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Other advantages and features of the present disclosure will be seen from reading the detailed description of implementations and embodiments, which are in no way limiting, and from the following accompanying drawings, in which: 
         FIG. 1  schematically shows a teletransmission system according to the present disclosure; 
         FIG. 2  schematically shows a first embodiment of the teletransmission method according to the present disclosure for a teleassistance operation; 
         FIG. 3  schematically shows a particular example of a scene of intervention carried out with the teletransmission method according to the present disclosure; 
         FIG. 4  schematically shows a second embodiment of the teletransmission method according to the present disclosure, in the context of intervention represented in  FIG. 3 ; 
         FIG. 5  schematically shows a third embodiment of the teletransmission method according to the present disclosure, including video stream substitutions; and 
         FIG. 6  schematically shows a particular application of a teletransmission system according to the present disclosure, on board an inspection drone. 
     
    
    
     DETAILED DESCRIPTION 
     Since the embodiments described hereinafter are in no way limiting, variants of the present disclosure may also be considered comprising only a selection of described characteristics, isolated from other described characteristics (even if this selection is isolated within a sentence comprising these other characteristics), if this selection of characteristics is sufficient to confer a technical advantage or to differentiate embodiments of the present disclosure with respect to the prior art. This selection comprises at least one characteristic, preferably functional without structural details, or with only one part of the structural details if this part alone is sufficient to confer a technical advantage or to differentiate embodiments of the present disclosure with respect to the prior art. 
     With reference to  FIG. 1 , a teletransmission system S comprises portable equipment P provided with a camera, a data processing unit, a screen having a graphic interface and a wireless communication module. This portable equipment can be produced from conventional devices, such as connected tablets or telescopes, or can be specifically designed for a given use. This portable equipment P incorporates software means designed to process a video stream captured by the camera. 
     A field operator (not shown) working on an intervention scene SI uses the portable equipment P to produce a video of this intervention scene and more particularly of a technical unit E located in this intervention scene SI. The video stream thus captured comprises both the unit E and a background B. 
     The teletransmission method according to the present disclosure is implemented in a software module included in the portable equipment P. This method is configured to modify the captured stream F in such a way that zones of interest I 1 , I 2 , I 3  visible on the unit E are deleted from the stream. The stream FM thus modified can then be transmitted by a communication network, not necessarily secured end to end, to a remote teleprocessing site ST. The modified stream FM thus received is then processed so as to be enriched, for example, by augmented reality techniques, then returned in the form of an enriched stream FE to the portable equipment P of the field operator. 
     With reference to  FIG. 2 , a description will now be provided of the different steps of the teletransmission method according to the present disclosure, implemented in the configuration of  FIG. 1 . 
     The stream F captured by the sensor C of the portable equipment P, potentially a camera, comprises both an image of the unit E including zones of interest I 1 , I 2 , I 3  and background elements B 1 , B 2 . 
     The zones of interest for which transmission is not desired can include, for example, a graphic design or a logo I 1  that represents a risk or a specific function. These zones of interest can also include labels or title blocks containing critical information I 2 , I 3 . 
     A first processing step T 1  is designed to identify the unit E within the scene of intervention and to extract it in order to obtain an extraction video stream FE with a neutral background NB no longer containing usable information. 
     Several methods of extraction in a video stream are already available in the state of the art. Document US 2013/0093788 A1, in particular, can be cited, which discloses a method for making a real object disappear in a display in mixed reality. 
     The neutralization of a subset of the image (background of the observed object, descriptive elements or others) consists of applying neutral colorimetric properties to the group(s) of pixels composing the subset. In other words, the properties of the pixels, generally defined by float values (example: RGB), are assigned the same color. In a second processing step T 2 , the zones of interest I 1 , I 2 , I 3  are identified then erased from the video stream FE in order to obtain a processed video stream FT in which the unit ET thus processed no longer contains zones of interest, which are replaced by erased zones EI 1 , EI 2  and EI 3 . 
     A video stream consists of a succession of images conventionally composed of pixels ordered in lines and columns. The identification of a unit known a priori within a video stream thus consists in determining, for each of the images composing the stream, one or more subsets of pixels corresponding to zones (in the sense of pixels) in which the unit appears. 
     The identification of a unit known a priori, such as zones of interest I 1 , I 2 , I 3 , can be done automatically and deterministically if the position of the camera at the origin of acquisition relative to the unit is known (in the spatial sense), as well as the intrinsic characteristics of the camera. 
     The position of the camera relative to the unit known a priori can be done through a model-based resetting consisting in determining the position according to an optimization approach intended, both locally and globally, to juxtapose the unit as observed and the geometric representation thereof known a priori. 
     Localization can also be done more trivially by locating the camera within the reference frame of an easily identifiable and locatable visual landmark (referring to datamatrix type markers), for which the position of the landmark is known within the reference frame of the unit to be identified (in the spatial sense). 
     Depending on their typology (background, identification plate, logo, or others), the zones of interest to be erased can be colorized in different ways. However, it can also be provided for these erased zones to appear as transparent in the processed video stream to be transmitted. 
     This stream thus processed—with a neutral background and zones of interest erased—is then transmitted (T 3 ). If an attempt at interception IN should succeed on the modified video stream FT during transmission, the informational harm would necessarily be reduced because the intercepted stream would be difficult to use to determine the location of the scene of intervention. 
     The remote processing site ST receives (T 4 ) this modified video stream representing the processed image ET of the unit E. The received video stream FR is then processed (T 5 ) in such a way that it is enhanced or enriched by addition of information A 1 , A 2  and instructions or orders IN. The video stream thus enhanced FA can then be transmitted (T 6 ) to the portable equipment P of the field operator. 
     With reference to  FIGS. 3 and 4  the teletransmission method according to the present disclosure can also be implemented to delete in the transmitted video stream any contextual information related to an intervention scene. 
     An intervention scene SI′ comprises, by way of nonlimiting example, equipment E′ installed in a room provided with two windows F 1 , F 2 . A large size object CT, placed in the vicinity of the unit E, causes a drop shadow O on the equipment E′ due to solar radiation through the first window F 1 . The second window F 2 , situated behind the unit E′, faces a characteristic landscape including, for example, a tree typical of the vegetation of the zone. Moreover, the unit E′ itself comprises two zones of interest I 4 , I 5  the transmission whereof is not desired. 
     The large size object CT can, for example, be provided with a label NU indicating the presence of radioactive components or materials. If the field operator decides to produce a video stream containing this object CT, this video stream will then be processed in such a way as to erase the contents of the previously identified label NU, in the video stream that will be teletransmitted. 
     The stream FI captured by the camera of the portable equipment P comprises both the drop shadow O and a view of the background landscape visible through the window F 2 . This shadow and this landscape could be used by an unauthorized third party intercepting the transmitted stream for the purpose of determining the location of the scene of intervention. It is therefore essential to avoid the transmission of this contextual information. 
     A first step (T 1 ′) contributes to extracting from the video stream FI′ the only video image of the unit E′, which then has a neutral background. The extraction stream FE′ is then processed (step T 2 ′) so as to erase the two zones of interest I 4 , I 5 . The stream thus processed FT′ then comprises a video image of the unit E′ comprising two erased zones EI 4 , EI 5 . 
     The function of a third step (T 3 ′) is to attenuate or modify the drop shadow O so as to generate a corrected video stream FC, which includes modified information about the solar lighting conditions, for example, by means of optical or digital filters designed to appreciably attenuate the shadow effects or the specularities, which would then become difficult to interpret for a third party who would intercept the video stream after processing. This corrected video stream FC, comprising substituted zones EI 4 , EI 5  and an attenuated shadow O′, can then be transmitted (step T 4 ′) by the communication network and received by the processing site ST in the form of a received video stream FR′. 
     With reference to  FIG. 5 , a third embodiment of the teletransmission method according to the present disclosure can also be provided in which the video stream actually transmitted no longer includes zones of interest but includes substituted zones and possibly a drop shadow substantially attenuated or modified by optical or digital filters. The desired objective is then to mislead an unauthorized third party who might intercept the transmitted video stream. 
     A field operator has captured a video stream FI″ by his portable equipment P, representing in a scene of intervention SI″ a unit E″ in front of a background B″, the unit E″ comprising, for example, two zones of interest I 6 , I 7 . 
     Processing steps T 1 ″, T 2 ″, T 3 ″—similar to those of T 1 ′, T 2 ′, T 3 ′ just described with reference to  FIG. 4 —respectively and successively enable extracting the unit E″ from its background, erasing the zones of interest I 6 , I 7  and processing by filtering of the drop shadow O on the unit E″. Thus, a processed video stream FT″ is obtained comprising a processed version O″ of the drop shadow. 
     With a view to compromising the exploitation of portions of conveyed images, several conventional infographic processes can be used. By working randomly on the properties of the images within a video stream (gamma, exposure, contrast, lighting) prior to their transmission, the ability to interpret them can be drastically limited with respect to the capture context (natural versus artificial lighting, orientation of the light, etc.). 
     In addition to neutralizing properties of the images, it is also conceivable to apply additional textures previously calculated on the basis of 3D renderings. Thus, drop shadows that are credible but not present in the observed reality can be embedded in the transmitted image, falsifying its interpretation by a potential interceptor. 
     A new processing step (T 4 ″) is designed to substitute fictional zones of interest IS 6 , IS 7 , for real zones of interest I 6 , I 7 . 
     For example, it can involve applying a color joined with a set of pixels for which the semantic information they carry is to be deleted. 
     Thus, a substituted video stream FS is obtained, which can then be transmitted (T 5 ″) by a communication network to the processing site ST, which receives a video stream FR″ including at the same time a modified image of the unit E″. In the event of undesirable interception during transmission of the substituted video stream FS, its use would lead to erroneous interpretations thus making it possible to protect the information sources about the actual location of the scene of intervention. 
     A teletransmission system according to the present disclosure can also be installed within a mobile platform P carried by an inspection drone D, as shown in  FIG. 6 . This drone D is, for example, provided for inspecting technical equipment SP having, on one or more of the faces thereof, zones of information I 8 , I 9  the transmission of which is undesirable because the interception thereof by third parties would be prejudicial. For example, zone I 8  can be a symbol indicating the presence of radioactive materials or components while zone I 9  can comprise data related to the origin of this technical equipment. 
     When the inspection drone D approaches the technical equipment SP, the camera C thereof captures a video stream corresponding to inspection objectives. A processor MP incorporated in the mobile platform P locally processes this video stream, identifies the zones of interest I 8 , I 9  as shapes corresponding to predetermined shapes and erases these zones of interest. The video stream thus processed FT 6 , including erased zones EI 8 , EI 9 , is then transmitted by a wireless transmission unit MT to a communication network R, then routed to a processing site ST. This drone can then be controlled by an operator working on site. 
     Of course, the present disclosure is not limited only to the examples that have just been described, and many other embodiments can be envisaged within the scope of the present disclosure. Thus, the equipment involved in the teletransmission operations implemented in the systems according to the present disclosure can be of any type. These teletransmission operations can also be implemented for purposes other than telesurveillance or inspection from mobile platforms.