Patent Description:
Remote operation of vehicles provides several benefits as the need of having a human present at or inside the vehicle is removed. As automation in the automotive and trucking industry increases there are many benefits of separating a driver from the vehicle. In trucking and logistics, drivers have legal requirements for taking breaks and hours they can work during the day. By separating the drivers from trucks, restrictions placed on drivers can be separated from the usability of the trucks themselves. Additionally, as automation of driving advances, there are times where an autonomous system is not capable of efficiently operating on its own and would benefit from human input. In this sense there are advantages to have vehicles with both an autonomous driving mode and a remote operated driving mode.

There are even further benefits such as in case the environment where the vehicle operates is inhospitable, as is the case for planetary rovers, bomb disarming vehicles and vehicles furnished with nuclear reactor maintenance devices, or simply due to distance.

In particular, remotely operated vehicles enable, for example, freight carrying vehicles which do not place a human driver at risk of a traffic accident.

Document <CIT> discloses a near real-time data and video streaming for a vehicle, robot or a drone, wherein video streams are transferred from the vehicles, robots or drones to a control centre device. Accelerated GPU video processing is used to reduce latencies associated with handling the video streams. Timestamps may be embedded in the image frames. Moreover, document <CIT> discloses a vehicular processor for an autonomous vehicle. The vehicular processor is arranged with Artificial Intelligence, AI, adapted to process images taken from a video camera to select some portions which are sent to a remote station for remote assistance. Timestamps may also be embedded in the image sent to the remote station.

According to some aspects, there is provided the subject-matter of the independent claims. Some embodiments are defined in the dependent claims.

According to a first aspect of the present disclosure, there is provided an apparatus comprising a configurable visual indicator and a video sensor arranged to generate a sensor data stream which is dependent on a state of the configurable visual indicator by capturing the configurable visual indicator, which is in a field of view of the video sensor, in the sensor data stream as a visual element, control circuitry configured to select the state of the configurable visual indicator, to process operating instructions received in the apparatus from a remote driving station, to provide to the remote driving station the sensor data stream and to at least one of: provide to the remote driving station an indication of the selected state of the configurable visual indicator, to enable the remote driving station to detect a malfunction in the sensor data stream, the indication distinct from the configurable visual indicator and observe the state of the configurable visual indicator, based on the sensor data stream, and determine whether the observed and the selected state of the configurable visual indicator are consistent, to detect a malfunction in the sensor data stream, wherein the apparatus is a remotely operated vehicle, or is configured to be installed in a remotely operated vehicle.

According to a second aspect of the present disclosure, there is provided an apparatus comprising at least one processing core, at least one memory including computer program code, and the computer program code being configured to, with the at least one processing core, cause the apparatus at least to perform as a remote driving station to a remotely operated vehicle by providing operating instructions to the remotely operated vehicle, receive from the remotely operated vehicle a sensor data stream, render the sensor data stream on a display wherein an image of a configurable visual indicator is visible on the display when the sensor data stream is rendered on the display and at least one of: receive an indication of a selected state of the configurable visual indicator in the remotely operated vehicle, provide a visual signal based on the indication of the selected state of the configurable visual indicator, to enable an operator to detect a malfunction in the sensor data stream, the indication being distinct from the image of the configurable visual indicator , and monitor the image of the configurable visual indicator using a video sensor, scan an output of the video sensor to determine a state of the image of the configurable visual indicator, and include the determined state of the image of the configurable visual indicator in information provided to the remotely operated vehicle.

According to a third aspect of the present disclosure, there is provided a method comprising operating, in an apparatus, a configurable visual indicator and a video sensor arranged to generate a sensor data stream which is dependent on a state of the configurable visual indicator by capturing the configurable visual indicator, which is in a field of view of the video sensor, in the sensor data stream as a visual element, selecting the state of the configurable visual indicator, processing operating instructions received in the apparatus from a remote driving station, providing to the remote driving station the sensor data stream and at least one of: firstly, providing to the remote driving station an indication of the selected state of the configurable visual indicator, to enable the remote driving station to detect a malfunction in the sensor data stream, the indication distinct from the configurable visual indicator and secondly, observing of the state of the configurable visual indicator, based on the sensor data stream, and determining whether the observed the selected state of the configurable visual indicator are consistent, to detect a malfunction in the sensor data stream, wherein the apparatus is a remotely operated vehicle, or is configured to be installed in a remotely operated vehicle.

According to a fourth aspect of the present disclosure, there is provided a method comprising performing, by an apparatus, as a remote driving station to a remotely operated vehicle by providing operating instructions to the remotely operated vehicle, receiving from the remotely operated vehicle a sensor data stream, rendering the sensor data stream on a display wherein an image of a configurable visual indicator is visible on the display when the sensor data stream is rendered on the display and at least one of: firstly, receiving an indication of a selected state of the configurable visual indicator in the remotely operated vehicle, providing a visual signal based on the indication of the selected state of the configurable visual indicator, to enable an operator to detect a malfunction in the sensor data stream, the indication being distinct from the image of the configurable visual indicator, and secondly, monitoring the image of the configurable visual indicator using a video sensor, scanning an output of the video sensor to determine a state of the image of the configurable visual indicator, and including the determined state of the image of the configurable visual indicator in information provided to the remotely operated vehicle.

According to a fifth aspect of the present disclosure, there is provided a non-transitory computer readable medium having stored thereon a set of computer readable instructions that, when executed by at least one processor, cause an apparatus to at least operate a configurable visual indicator and a video sensor arranged to generate a sensor data stream which is dependent on a state of the configurable visual indicator by capturing the configurable visual indicator, which is in a field of view of the video sensor, in the sensor data stream as a visual element, select the state of the configurable visual indicator, process operating instructions received in the apparatus from a remote driving station, provide to the remote driving station the sensor data stream and at least one of: firstly, provide to the remote driving station an indication of the selected state of the configurable visual indicator, to enable the remote driving station to detect a malfunction in the sensor data stream, the indication distinct from the configurable visual indicator and secondly, observe the state of the configurable visual indicator, based on the sensor data stream, and determine whether the observed and the selected state of the configurable visual indicator are consistent, to detect a malfunction in the sensor data stream, wherein the apparatus is a remotely operated vehicle, or is configured to be installed in a remotely operated vehicle.

According to a sixth aspect of the present disclosure, there is provided a non-transitory computer readable medium having stored thereon a set of computer readable instructions that, when executed by at least one processor, cause an apparatus to at least perform as a remote driving station to a remotely operated vehicle by providing operating instructions to the remotely operated vehicle, receive from the remotely operated vehicle a sensor data stream, render the sensor data stream on a display wherein an image of a configurable visual indicator is visible on the display when the sensor data stream is rendered on the display and at least one of: firstly, receive an indication of a selected state of the configurable visual indicator in the remotely operated vehicle, provide a visual signal based on the indication of the selected state of the configurable visual indicator, to enable an operator to detect a malfunction in the sensor data stream, the indication being distinct from the image of the configurable visual indicator, and secondly, monitor the image of the configurable visual indicator using a video sensor, scan an output of the video sensor to determine a state of the image of the configurable visual indicator, and include the determined state of the image of the configurable visual indicator in information provided to the remotely operated vehicle.

Using methods disclosed herein, dependability of remote operation may be enhanced by including in a sensor data stream originating in a remotely controlled vehicle information enabling checking, if the sensor data stream, such as a video stream, for example, is experiencing issues. Examples of issues include freezing, delay, quality and jitter. Three embodiments are disclosed, which may be used separately or in any combination in the same remotely operated vehicle, for example relating to distinct sensor data streams originating in the vehicle. Additionally, elements and aspects from the embodiments can be exchanged and combined between the embodiments.

<FIG> illustrates an example system in accordance with at least some embodiments of the present invention. Remotely operated vehicle <NUM> is at least in part operated remotely from remote driving station <NUM>. Remotely operated vehicle <NUM> will be referred to herein as vehicle <NUM> for the sake of brevity. By being remotely operated it is herein meant that at least a part of the control of vehicle <NUM> takes place from remote driving station <NUM>. There may be humans in vehicle <NUM>, including a human participating in controlling vehicle <NUM>, but vehicle <NUM> is configured to be responsive concerning at least a part of its control inputs to operating instructions received in vehicle <NUM> from remote driving station <NUM>. For example, vehicle <NUM> may have a brake that a human in vehicle <NUM> may trigger to cause the vehicle to stop, with other control aspects such as speed, steering and operation of headlights being responsive to the operating instructions received from remote driving station <NUM>. In other words, vehicle <NUM> may be partly controlled locally from vehicle <NUM> and partly from remote driving station <NUM>. Vehicle <NUM> may also have no humans aboard at all and/or vehicle <NUM> may be completely controlled from remote driving station <NUM>.

Vehicle <NUM> may comprise a car, such as a taxi, a van, a truck or a tractor unit for a trailer. Alternatively, vehicle <NUM> may comprise hazardous-purpose vehicle such as a nuclear reactor maintenance buggy or bomb disposal vehicle.

As vehicle <NUM> is mobile, it may be configured to communicate with remote driving station <NUM> using, at least partly, wireless communication. In the example of <FIG>, vehicle <NUM> is furnished with a cellular communication module configured to participate in a wireless link <NUM> with base station <NUM>. Wireless link <NUM> and base station <NUM> are configured to function based on a cellular communication standard, such as long term evolution, LTE, or fifth generation, <NUM>, as specified by the <NUM>rd generation partnership project, 3GPP. An advantage of cellular systems is their great range as they can provide an uninterrupted connection to a vehicle roaming even internationally. On the other hand if less mobility is needed, a non-cellular wireless communication system may be used, such as wireless local area network, WLAN, or worldwide interoperability for microwave access, WiMAX. In some embodiments, satellite-based communication may be employed additionally or alternatively to terrestrial communication links, such as cellular or non-cellular links. Networks <NUM> and <NUM> may, in some embodiments, be traversed by a communication path between vehicle <NUM> and remote driving station <NUM>.

In some embodiments, instead of a wireless connection, a wired connection between vehicle <NUM> and remote driving station <NUM> may be used. For example, in case the range of mobility of vehicle <NUM> is limited, a wired connection may be appropriate. Advantages of a wired connection include fast datarate, low latency and low risk of an interrupted connection. In some embodiments with limited range of vehicle <NUM>, a direct wireless connection from remote driving station <NUM> to vehicle <NUM> may be used.

As a vehicle may move around on public streets, or in other embodiments interact with hazardous materials, dependability of remote operation is of considerable interest. In detail, a sensor data stream, such as a video stream, from vehicle <NUM> should have sufficient quality to enable a remote driver or remote driver system to recognize traffic signs and pedestrians, for example. Likewise, the sensor data stream should have a latency sufficiently low to enable fast reactions from remote driving station <NUM> to surprising developments in the driving environment. In general, remote driving station <NUM> may house a human remote driver, and/or an automated, at least partially autonomous driving algorithm. In some embodiments, a human remote driver is assisted by a driving algorithm.

<FIG> illustrates an example system in accordance with at least some embodiments of the present invention. The figure illustrates vehicle <NUM> and remote driving station <NUM> of <FIG> in more detail. Vehicle <NUM> comprises control circuitry <NUM>, a configurable visual indicator <NUM>, such as a light-emitting diode, LED, or another kind of light source, streamer <NUM> and video sensor <NUM>. Video sensor <NUM> may comprise, for example, a video camera arranged to image a view through a windscreen of vehicle <NUM>, as well as configurable visual indicator <NUM>. Remote driving station <NUM> comprises a player <NUM>, a display <NUM>, controller <NUM> and configurable visual indicator <NUM>. Configurable visual indicator <NUM> may comprise, alternatively or additionally to a light source, a mechanical element, such as windscreen wiper, arranged to provide the visual indication by changing its place and/or orientation, to be imaged by video sensor <NUM>. Yet further, the configurable visual indicator <NUM> may comprise a mechanism that controls the orientation of video sensor <NUM> itself, for example by turning video sensor <NUM> from a first orientation to a second orientation, and back.

In use, control circuitry <NUM> controls a state of configurable visual indicator <NUM> by switching it between states, such as on and off, for example according to a random, pseudo-random, or deterministic but specially crafted pattern. In other words, control circuitry <NUM> may be configured to repeatedly select the state of configurable visual indicator <NUM>. As configurable visual indicator <NUM> is within the field of view of video sensor <NUM>, the state of configurable visual indicator <NUM> is captured into a sensor data stream generated by video sensor <NUM> as a visual element, along with other objects within the field of view of video sensor <NUM>.

Streamer <NUM> provides the sensor data stream to remote driving station <NUM>, such that streamer provides to remote driving station <NUM> also an indication of the selected state of the configurable visual indicator, for example as metadata of the sensor data stream, or otherwise in connection with providing the sensor data stream.

At the remote driving station, player <NUM> controls display <NUM> based on the sensor data stream to render on display <NUM> the view imaged by video sensor <NUM>, such that an image 154a of configurable visual indicator <NUM> is visible on display <NUM> when the sensor data stream is rendered there. Controller <NUM> separately controls configurable visual indicator <NUM> based on the indication of the selected state of configurable visual indicator <NUM>, such that when the connection between vehicle <NUM> and remote driving station <NUM> is functioning well and the sensor data stream is not frozen, configurable visual indicator <NUM> and image 154a should have the same, or corresponding, state. An example of a corresponding state is one where changes in states of image 154a and configurable visual indicator <NUM> satisfy a delay requirement. Operating instructions are provided from remote driving station <NUM> to vehicle <NUM> to remotely operate the vehicle.

An operator of remote driving station <NUM> may, when remotely operating vehicle <NUM>, detect problems with the sensor data stream when image 154a and configurable visual indicator <NUM> have different states. Vehicle <NUM> may then be instructed to stop, for example.

More specifically, control circuitry <NUM> may provide to streamer <NUM> an identity of vehicle <NUM> and/or a timestamp in addition to the indication of the selected state of configurable visual indicator <NUM>. These data may be provided in a data structure, which may be input to a checksum algorithm, the resulting checksum also being provided to streamer <NUM>, for delivery to remote driving station <NUM>, for example embedded as metadata in the sensor data stream. Remote driving station <NUM> may extract the identity and/or timestamp, and provide a copy or copies of these back to vehicle <NUM> in connection with providing the operating instructions for implementing the remote driving. Control circuitry <NUM> may then check that the identity received from remote driving station <NUM> matches the identity of vehicle <NUM> provided to the remote driving station, and/or that the timestamp received is not too old compared to the present time. These checks enable detecting error scenarios where vehicle <NUM> is provided operating instructions of another vehicle (identity), and/or where communications between vehicle <NUM> and remote driving station <NUM> has excessive delay (timestamp). In either error case, vehicle <NUM> may be configured to stop, for example. Both remote driving station <NUM> and vehicle <NUM> may verify checksums calculated over information communicated to them by the other party, to detect bit errors in the information. Remote driving station <NUM> may also be configured to provide the indication of the selected state of configurable visual indicator <NUM> back to vehicle <NUM>, for example together with the operating instructions.

These mechanisms are capable of detecting freezing and significant delays of the sensor data stream, be these due to video sensor <NUM>, streamer <NUM>, player <NUM>, display <NUM> or the connection between vehicle <NUM> and remote driving station <NUM>. Tampering of the connection may be detected using normal cryptographic methods between vehicle <NUM> and remote driving station <NUM>. Bit errors in communicated information may be detected from the verification of checksums. As noted, sending operating instructions to a wrong vehicle can also be prevented from influencing actual operation of vehicle <NUM>.

<FIG> illustrates an example system in accordance with at least some embodiments of the present invention. The system of <FIG> resembles that of <FIG>, an indeed like numbering denotes like structure as in <FIG>. Different from <FIG>, remote driving station <NUM> comprises, instead of controller <NUM> and configurable visual indicator <NUM>, a scanner <NUM> and video sensor <NUM>. Video sensor <NUM> is configured to monitor image 154a of configurable visual indicator <NUM>. In use, control circuitry <NUM> controls a state of configurable visual indicator <NUM> by switching it on and off, for example according to a random or pseudo-random pattern, as in <FIG>. The sensor data stream from vehicle <NUM> is rendered in remote driving station <NUM> to display <NUM>, as was the case in the system of <FIG>.

In the solution of <FIG>, remote driving station <NUM> scans an output of video sensor <NUM> to determine, what is the state of image 154a. This state is then included in the information provided back to vehicle <NUM>, for example in connection with providing the operating instructions to vehicle <NUM>. Vehicle <NUM> then compares the state received from remote driving station <NUM> to the state selected by control circuitry <NUM> for configurable visual indicator <NUM>. Thus vehicle <NUM> is enabled to detect if the two-way communication between itself and remote driving station <NUM> is sufficiently fast and delay-free for dependable and safe remote control of vehicle <NUM>. If there is a mismatch between the selected state of configurable visual indicator <NUM> and the state received from remote driving station <NUM>, vehicle <NUM> may be configured to stop. Advantageously in the system of <FIG>, a human operator need not remain vigilant to detect the mismatch, as was the case in the system of <FIG>.

In some embodiments, vehicle <NUM> is configured to measure a round-trip delay in communication between vehicle <NUM> and remote driving station <NUM> by changing the selected state of configurable visual indicator <NUM>, and then determining how much time elapses before the new selected state is present in the state received back from remote driving station <NUM>. This delay value may be provided to remote driving station <NUM>, which may display it to a human operator to enhance his awareness of the remote operation situation. For example, responsive to an increase in the delay value, the human operator may decrease a maximum speed of vehicle <NUM>, to give himself more time to respond to changes in the driving environment. When remote driving station <NUM> does not have a human operator, the delay value may be provided to a remote driving algorithm, which may use it as one of its inputs. For example, the remote driving algorithm may reduce the speed of vehicle <NUM> as a response to the delay increasing, to provide more time to respond to changes in the driving environment.

Compared to the system of <FIG>, the system of <FIG> provides the effect that vehicle <NUM> is enabled to respond to the delay value determined between itself and remote driving station <NUM>. Further, no human is needed in the loop and measurement of the delay value is enabled, as described above. Freezing and significant delays of the video stream, be it in the video sensor, streamer, player or display, will be detectable by the vehicle's check. As was the case in <FIG>, vehicle <NUM> may provide to remote driving station <NUM>, for example as metadata in the sensor data stream, an identity of vehicle <NUM> and/or a timestamp, which remote driving station <NUM> provides back to vehicle <NUM> in connection with providing the operating instructions, to enable detection of operating instructions sent to a wrong vehicle, and/or determination of whether the timestamps received in return from remote driving station <NUM> are too old.

<FIG> illustrates an example system in accordance with at least some embodiments of the present invention. The system of <FIG> resembles that of <FIG>, an indeed like numbering denotes like structure as in <FIG>. In use, control circuitry <NUM> controls a state of configurable visual indicator <NUM> by switching it on and off, for example according to a random or pseudo-random pattern.

In the system of <FIG>, the sensor data stream from video sensor <NUM> is scanned in scanner <NUM> of vehicle <NUM>, to detect the state of configurable visual indicator <NUM> in the sensor data stream in vehicle <NUM> itself, without involving remote driving station <NUM> in the detection. Scanner <NUM> provides the scanned state to control circuitry <NUM>, to enable control circuitry <NUM> to detect error states in the sensor data stream locally in vehicle <NUM>, before the sensor data stream is provided to remote driving station <NUM>. In case there is a divergence between the selected state of configurable visual indicator <NUM> and the detected state of configurable visual indicator <NUM> in the sensor data stream, vehicle <NUM> may be configured to stop, for example, as the sensor data stream is unreliable. Although illustrated in <FIG> as traversing streamer <NUM>, the connection between scanner <NUM> and video sensor <NUM> need not traverse streamer <NUM> and may be direct.

Vehicle <NUM> provides to remote driving station <NUM> the sensor data stream, which is modified in its payload content to include embedded as a visual element in the stream a nonce selected by vehicle <NUM>, selected for example by control circuitry <NUM>. The nonce may be changed several times during use, for example according to a random or pseudo-random pattern. At the remote driving station, the nonce NCE is rendered on display <NUM> as a visual element from the sensor data stream, and captured by video sensor <NUM> of the remote driving station. The nonce from video sensor <NUM> is scanned in scanner <NUM> and provided back to vehicle <NUM>, to enable vehicle <NUM> to monitor the quality of the connection between itself and the remote driving station, as was the case in <FIG> and the state of the configurable visual indicator <NUM>. Vehicle <NUM> may store a sequence of past nonces, to enable determining a length of communication delay in case the nonce received from remote driving station <NUM> does not match the current nonce.

As was the case in <FIG> and <FIG>, vehicle <NUM> may provide to remote driving station <NUM>, for example as metadata in the sensor data stream or otherwise associated with the sensor data stream, an identity of vehicle <NUM> and/or a timestamp, which remote driving station <NUM> provides back to vehicle <NUM> in connection with providing the operating instructions, to enable detection of operating instructions sent to a wrong vehicle, and/or determination of whether the timestamps received in return from remote driving station <NUM> is too old. Checksums may be used in both directions to detect bit errors in the communicated information, also like the systems of <FIG> and <FIG>.

In general concerning <FIG>, <FIG> and <FIG>, the process of remote control of vehicle <NUM> may proceed in a continuous manner, where the sensor data stream from vehicle <NUM> is continuously provided to remote driving station <NUM>, and the operating instructions likewise provided as a continuous stream from remote driving station <NUM> to vehicle <NUM>. Further in general, the elements described above in the systems of <FIG>, <FIG> and <FIG> may be implemented as hardware, software or combinations of hardware and software. Further, certain ones of the elements illustrated as distinct in the figures for the same of clarity may be implemented as a single physical integrated chip and/or software executable, as applicable.

<FIG> illustrates an example apparatus capable of supporting at least some embodiments of the present invention. Illustrated is device <NUM>, which may comprise, for example, in applicable parts, vehicle <NUM> or a controller thereof of <FIG>, or the remote driving station <NUM> or a controller thereof. Comprised in device <NUM> is processor <NUM>, which may comprise, for example, a single- or multi-core processor wherein a single-core processor comprises one processing core and a multi-core processor comprises more than one processing core. Processor <NUM> may comprise, in general, a control device. Processor <NUM> may comprise more than one processor. Processor <NUM> may be a control device. A processing core may comprise, for example, a Cortex-A8 processing core manufactured by ARM Holdings or a Zen processing core designed by Advanced Micro Devices Corporation. Processor <NUM> may comprise at least one AMD Opteron and/or Intel Core processor. Processor <NUM> may comprise at least one application-specific integrated circuit, ASIC. Processor <NUM> may comprise at least one field-programmable gate array, FPGA. Processor <NUM> may be means for performing method steps in device <NUM>. Processor <NUM> may be configured, at least in part by computer instructions, to perform actions.

Device <NUM> may comprise a transmitter <NUM>. Device <NUM> may comprise a receiver <NUM>. Transmitter <NUM> and receiver <NUM> may be configured to transmit and receive, respectively, information in accordance with at least one suitable communication standard. Transmitter <NUM> may comprise more than one transmitter. Receiver <NUM> may comprise more than one receiver.

Device <NUM> may comprise user interface, UI, <NUM>. UI <NUM> may comprise at least one of a display, a keyboard, a touchscreen, a vibrator arranged to signal to a user by causing device <NUM> to vibrate, a speaker and a microphone. A user may be able to operate device <NUM> via UI <NUM>, for example to configure remote operation parameters.

Device <NUM> may comprise further devices not illustrated in <FIG>. In some embodiments, device <NUM> lacks at least one device described above. For example, some devices <NUM> may lack a NFC transceiver <NUM>.

Processor <NUM>, memory <NUM>, transmitter <NUM>, receiver <NUM>, NFC transceiver <NUM> and/or UI <NUM> may be interconnected by electrical leads internal to device <NUM> in a multitude of different ways. For example, each of the aforementioned devices may be separately connected to a master bus internal to device <NUM>, to allow for the devices to exchange information. However, as the skilled person will appreciate, this is only one example and depending on the embodiment various ways of interconnecting at least two of the aforementioned devices may be selected without departing from the scope of the present invention.

<FIG> illustrates signalling in accordance with at least some embodiments of the present invention. Two separate processes A and B are illustrated. On the vertical axes are disposed, on the left, vehicle <NUM> of <FIG>, <FIG> and <FIG> and its control circuitry <NUM> and streamer <NUM>, and on the right, remote driving station <NUM> of <FIG>, <FIG> and <FIG> and its player <NUM>, scanner <NUM> and video sensor <NUM>. Time advances from the top toward the bottom.

In process A, corresponding to <FIG>, in phase <NUM> control circuitry <NUM> provides to streamer <NUM> metadata for the sensor data stream of vehicle <NUM>. Streamer <NUM>, in the example of <FIG>, embeds the metadata received in phase <NUM> as metadata of the sensor data stream from video sensor <NUM>, and provides the sensor data stream to remote driving station <NUM>, to player <NUM>, phase <NUM>. Player <NUM> renders the stream onto the display of remote driving station, which video sensor <NUM> is arranged to image, as described herein above. Continuing the process, video sensor <NUM> provides its sensor data stream to scanner <NUM>, and player <NUM> provides the metadata of the sensor data stream received from vehicle <NUM> to scanner <NUM> as well, phase <NUM>. Scanner <NUM> determines, phase <NUM>, based on the sensor data stream from video sensor <NUM>, which state configurable visual indicator <NUM> has in the sensor data stream received from vehicle <NUM>, and provides it to control circuitry <NUM> of vehicle <NUM> in phase <NUM>. Phase <NUM> may also comprise the provision of operating instructions for vehicle <NUM> from remote driving station <NUM>. In phase <NUM>, control circuitry <NUM> of vehicle <NUM> determines whether the state of configurable visual indicator <NUM> as received from remote driving station <NUM> in phase <NUM> is the same as the one selected in control circuitry <NUM>, or at least within a tolerable delay of the selected one. As a variant, the metadata of phase <NUM> may proceed to the display of remote driving station <NUM>, and be read from there by video sensor <NUM>, to then be provided to scanner <NUM>.

As described above, in some embodiments the metadata provided for the sensor data stream of vehicle <NUM>, phase <NUM>, comprises an identity of vehicle <NUM> and/or a timestamp assigned by control circuitry <NUM>. In these embodiments, the identity of vehicle <NUM> and/or the timestamp are provided back to vehicle <NUM> in phase <NUM>, to enable control circuitry <NUM> to check the operating instructions are for vehicle <NUM> and not another vehicle, and/or that the timestamp is sufficiently fresh to guarantee the round-trip communication delay is not excessive. In some embodiments where the identity of vehicle <NUM> and/or the timestamp are not provided from vehicle <NUM>, the metadata of phases <NUM> and <NUM> may be absent.

In process B, corresponding to <FIG>, in phase <NUM> control circuitry <NUM> provides to streamer <NUM> metadata for the sensor data stream of vehicle <NUM>, the metadata comprising a nonce. In phase <NUM> streamer <NUM> provides the sensor data stream of the video sensor of vehicle <NUM> to scanner <NUM>, which detects the state of configurable visual indicator <NUM> from the sensor data stream of the video sensor of vehicle <NUM>, and provides, phase <NUM>, this state to control circuitry <NUM>, to enable the control circuitry to check, whether the sensor data stream functions adequately inside vehicle <NUM>, as described herein above.

In phase <NUM>, streamer <NUM> provides the sensor data stream of vehicle <NUM>, into which the nonce has been embedded as a visual item in the payload, to remote driving station <NUM>, in particular to player <NUM>, which renders the sensor data stream onto the display of remote driving station <NUM>. Player <NUM> provides metadata of the sensor data stream, if any, to scanner <NUM> in phase <NUM>, and video sensor <NUM> provides its sensor data stream to scanner <NUM> in phase <NUM>. In phase <NUM>, scanner <NUM> determines the nonce by scanning, that is digitizing and image-recognizing, at least a part of the sensor data stream from video sensor <NUM>. In phase <NUM>, the nonce obtained by scanner <NUM> from the sensor data stream of video sensor <NUM> is provided to vehicle <NUM> along with operating instructions for vehicle <NUM>. In phase <NUM>, control circuitry <NUM> may check it is the same nonce as the one provided to streamer <NUM> in phase <NUM>, or at least a nonce issued within a tolerable delay to the most recently issued nonce. As a variant, the metadata of phase <NUM> may proceed to the display of remote driving station <NUM>, and be read from there by video sensor <NUM>, to then be provided to scanner <NUM>.

As described above, in some embodiments the metadata provided for the sensor data stream of vehicle <NUM>, phase <NUM>, comprises an identity of vehicle <NUM> and/or a timestamp assigned by control circuitry <NUM>. In these embodiments, the identity of vehicle <NUM> and/or the timestamp are provided back to vehicle <NUM> in phase <NUM>, to enable control circuitry <NUM> to check the operating instructions are for vehicle <NUM> and not another vehicle, and/or that the timestamp is sufficiently fresh to guarantee the round-trip communication delay is not excessive. Likewise in these embodiments, the timestamp and/or identity of the vehicle are provided directly from player <NUM> to scanner <NUM>, rather than via the display. Control circuitry <NUM> may vary the nonce in a random or pseudorandom manner, for example, in a timescale which enables characterizing the round-trip delay in communications between vehicle <NUM> and remote driving station <NUM>.

<FIG> is a flow graph of a method in accordance with at least some embodiments of the present invention. The phases of the illustrated method may be performed in vehicle <NUM>, or in a control device configured to control the functioning thereof, when installed therein.

Phase <NUM> comprises operating, in an apparatus, a configurable visual indicator and a video sensor arranged to generate a sensor data stream which is dependent on a state of the configurable visual indicator. Phase <NUM> comprises selecting the state of the configurable visual indicator, processing operating instructions received in the apparatus from a remote driving station, providing to the remote driving station the sensor data stream and at least one of: firstly, phase 530a, providing to the remote driving station an indication of the selected state of the configurable visual indicator, to enable the remote driving station to detect a malfunction in the sensor data stream, and secondly, phase 530b, obtaining an observation of the state of the configurable visual indicator, based on the sensor data stream, and determining whether the observation and the selected state of the configurable visual indicator are consistent, to detect a malfunction in the sensor data stream, wherein the apparatus is a remotely operated vehicle, or is configured to be installed in a remotely operated vehicle.

One skilled in the relevant art will recognize, however, that the invention can be practised without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the invention.

Claim 1:
An apparatus (<NUM>) characterised in that it comprises:
- a configurable visual indicator (<NUM>) and a video sensor (<NUM>) arranged to generate a sensor data stream which is dependent on a state of the configurable visual indicator (<NUM>) by capturing the configurable visual indicator (<NUM>), which is in a field of view of the video sensor (<NUM>), in the sensor data stream as a visual element;
- control circuitry (<NUM>) configured to select the state of the configurable visual indicator (<NUM>), to process operating instructions received in the apparatus (<NUM>) from a remote driving station (<NUM>), to provide to the remote driving station (<NUM>) the sensor data stream and to at least one of:
▪ provide to the remote driving station (<NUM>) an indication of the selected state of the configurable visual indicator (<NUM>) to enable the remote driving station (<NUM>) to detect a malfunction in the sensor data stream, the indication distinct from the configurable visual indicator (<NUM>), and
▪ observe the state of the configurable visual indicator (<NUM>), based on the sensor data stream, and determine whether the observed and the selected state of the configurable visual indicator (<NUM>) are consistent, to detect a malfunction in the sensor data stream,
- wherein the apparatus is a remotely operated vehicle (<NUM>), or is configured to be installed in a remotely operated vehicle.