IMAGE FRAME STREAMING SYTEM

An image frame streaming system may include a vehicle, a video camera carried by the vehicle to output image frames, a multi-media framework carried by the vehicle and configured to generate a first stream and a second stream from the image frames, a first consumer to receive the first stream and a second consumer to receive the second stream. The first consumer and the second consumer are each selected from a group of consumers consisting of: a neural network; a live video stream presenter; and a historical time clip storage.

BACKGROUND

Vehicles, ground and airborne, come in a variety of forms such as tractors, semi-trailers, cement mixers, trucks, harvesters, sprayers, construction vehicles, drones, planes, helicopters and the like. Such vehicles may have a variety of operational parameters and may conduct a variety of different operations. Managing and analyzing their operations is sometimes challenging, but may offer opportunities for enhanced automation, land, vineyard, crop or orchard management, vehicle maintenance, operator assignment, vehicle design, and vehicle use.

DETAILED DESCRIPTION OF EXAMPLES

Disclosed are example image frame streaming systems, image frame streaming vehicles and image frame streaming methods that facilitate sub second latency camera frame transmission. The example systems, vehicles and methods facilitate such sub second latency camera frame transmission for multiple cameras situated on a vehicle. The example systems, vehicles and methods utilize a multimedia framework that supports hardware encoding to compress live resolution frames or streams. Such streams have low latency and are optimized for adaptive bandwidth control. As result, there is minimal loss of frames, and the quality of the live feed is maintained throughout.

The example image frame streaming systems, image frame streaming vehicles and image frame streaming methods use a multimedia framework carried by the vehicle to generate multiple streams from image frames received from a video camera of the vehicle. In some implementations, all of the streams are formed from the same set of image frames. In other words, the source image frames used for generating each of the streams is identical. The streams are transmitted to at least two different consumers, wherein the consumers consist of a neural network, a live video stream presenter and/or a historical time clip storage.

In particular examples, a WebRTC framework is used. In particular examples, a Janus-Gateway provides the WebRTC solution. The Janus-Gateway may be installed on the vehicle server. On the other side of the Web app and Mobile app, a HTTP request may be sent to the Janus-Gateway to establish connection between the vehicle server and the mobile app on the operator's browser. In such examples, a relay server or a Traversal Using Relays around NAT (TURN) server is employed. In some implementations, the TURN server may be developed using Amazon Elastic Computer Cloud (AWS).

For purposes of this application, the term “processing unit” shall mean a presently developed or future developed computing hardware that executes sequences of instructions contained in a non-transitory memory. Execution of the sequences of instructions causes the processing unit to perform steps such as generating control signals. The instructions may be loaded in a random-access memory (RAM) for execution by the processing unit from a read only memory (ROM), a mass storage device, or some other persistent storage. In other embodiments, hard wired circuitry may be used in place of or in combination with software instructions to implement the functions described. For example, a controller may be embodied as part of one or more application-specific integrated circuits (ASICs). Unless otherwise specifically noted, the controller is not limited to any specific combination of hardware circuitry and software, nor to any particular source for the instructions executed by the processing unit.

For purposes of this disclosure, unless otherwise explicitly set forth, the recitation of a “processor”, “processing unit” and “processing resource” in the specification, independent claims or dependent claims shall mean at least one processor or at least one processing unit. The at least one processor or processing unit may comprise multiple individual processors or processing units at a single location or distributed across multiple locations.

For purposes of this disclosure, the phrase “configured to” denotes an actual state of configuration that fundamentally ties the stated function/use to the physical characteristics of the feature proceeding the phrase “configured to”.

For purposes of this disclosure, unless explicitly recited to the contrary, the determination of something “based on” or “based upon” certain information or factors means that the determination is made as a result of or using at least such information or factors; it does not necessarily mean that the determination is made solely using such information or factors. For purposes of this disclosure, unless explicitly recited to the contrary, an action or response “based on” or “based upon” certain information or factors means that the action is in response to or as a result of such information or factors; it does not necessarily mean that the action results solely in response to such information or factors.

FIG.1is a diagram schematically illustrating portions of an example image frame streaming system500. System500comprises vehicle520, video camera522, multimedia framework530, and consumers550-1,550-2(collectively referred to as consumers550). Vehicle520comprises a vehicle in the form of a tractor, semi-trailer, cement mixer, truck, harvester, sprayer, construction vehicle, drone, plane, helicopter and the like, or any other vehicle for which cameras are provided to provide image frames for image frame streaming and other uses.

Video camera522comprise a camera carried by vehicle520configured to output image frames. Video camera522may be in the form of a 3D or stereo camera or a two-dimensional camera. In some implementations, vehicle520may comprise multiple video cameras522.

Multimedia framework530is carried by vehicle520and is configured to generate image streams534-1and534-2(collectively referred to as image streams534) from the image frames523received from video camera522. In some implementations, multimedia framework530comprises a GStreamer pipeline-based multimedia framework. Streams534-1and534-2are transmitted to consumers550-1and550-2, respectively.

Consumers550utilize streams534of images to carry out various tasks. Each of consumers550is selected from a group of consumers consisting of: a neural network, a live video stream presenter and a historical time clip storage. In some implementations, the neural network utilizes the stream of images or image frames to train a deep learning network and/or to analyze such images using an already trained network to carry out automated control of vehicle520. In some implementations, the live video stream presenter provides a live sub second latency camera frame transmission to an operator, residing either locally on vehicle520or at a remote location. In some implementations, the historical time clip storage facilitates recording and storage of image frames for subsequent viewing and analysis.

In some implementations, the live video stream presenter may comprise a web real time communications (RTC) server carried by the vehicle. In some implementations, the web RTC server may comprise a Janus Gateway server. In some implementations, the live stream video presenter may comprise a cloud-based media server that is to receive a first stream from the web RTC server and a live stream display device to receive the first stream from the media server and display the live stream to a person.

In some implementations, the historical time clip storage may comprise a video clip up loader carried by the vehicle and configured to receive a video clip based upon the stream. The historical time clip storage may further comprise a cloud-based storage configured to receive the video clip from the video clip up loader and a video clip display device to receive the video clip from the cloud-based storage and display the video clip to a person.

FIG.2is a diagram schematically illustrating portions of an example image frame streaming system600. System600is similar to system500except that system800specifically comprises image frame stream consumers650-1,650-2and650-3(collectively referred to as consumers650) in place of consumers550. Those remaining components of system600, which correspond to components of system700are numbered similarly and are described above with respect to system700. As shown byFIG.2, multimedia framework530is configured to generate streams634-1,634-2and634-3(collectively referred to as streams634) from image frame523received from video cameras522and which are transmitted to consumers650-1,650-2and650-3, respectively.

Consumer650-1comprises a neural network (also referred to as machine learning). The neural network utilizes the stream of images or image frames to train a deep learning network and/or to analyze such images using an already trained network to carry out automated control of vehicle520.

Consumer650-2comprises a historical time clip storage (HTCS). The historical time clip storage650-2comprises a video clip up loader860, a cloud-based storage662and a display664. Video clip uploader660is configured to receive or generate a video clip from stream634-2. Uploader660is further configured to upload the video clip to cloud-based storage662. Display864comprises a display in communication with the cloud-based storage662and is configured to receive the video clip from the cloud-based storage662(a remote server) and to present the video clip to a person. Display664may be local, carried by vehicle520, or may be remote for presenting the video clip to a remote operator or manager of vehicle520.

Live video stream presenter650-3receives stream634-3and presents a sub second latency video stream to a person such that the person can see what the camera is seeing, live or in real time. Live video stream presenter650-3may include a display for presenting the live video stream. The display may be the same as display664or may be a separate display. The display may be one that is located on vehicle520or one that is remote from vehicle520.

FIG.3is a flow diagram of an example image frame streaming method700and may be carried out by systems500,600or systems described hereafter. As indicated by block704, a camera carried by a vehicle, such as vehicle520, captures image frames. The camera may be similar to video cameras522described above.

As indicated by block706, a multimedia framework generates a first stream and a second stream from the image frames captured in block704. As indicated block708, the first stream is transmitted to one of a neural network, a live video stream presenter and a historical time clip storage. As indicated by block710, the second stream is transmitted to another of the neural network, the live video stream presenter and the historical time clip storage.

In some implementations, a computer associated with the camera transmits the first stream to the neural network and transmits a second stream to a web real time communication (RTC server) which then transmits the second stream to a cloud-based live video stream server, wherein the live video stream or the video clip on the cloud-based server may be transmitted/downloaded to a web interface of an operator/manager. In some implementations, the computer associated with the camera further transmits a third stream to a video up loader which uploads video clips to the cloud-based historical time clip storage, wherein the video clips may be transmitted/downloaded to the web interface of the operator/manager.

FIG.4is a diagram illustrating portions of an example image frame streaming system800.FIG.4illustrates an example of how a vehicle may be provided with multiple cameras and associated computers to concurrently transmit streams of image frames to a neural network, an up loader for uploading video clips to a cloud-based video clip storage and to a web RTC server for transmitting live video streams to a cloud-based server for subsequent downloading or transmission to a web interface of an operator/manager.FIG.4illustrates how web RTC servers and uploaders may be shared amongst multiple cameras and computers on a vehicle and how a cloud service may be utilized to provide an operator/manager with live streams and access to stored video clips using a web interface. Image frame streaming system800comprises vehicle820, video cameras822-1,822-2,822-3, and822-4(collectively referred to as cameras822), computing devices or computers824-1,824-2,824-3and824-4(collectively referred to as computers824), up loaders826-1,826-2(collectively referred to as up loaders826), web RTC servers828-1,828-2(collectively referred to as servers828), smart screen831, neural networks840-1,840-2(collectively referred to as neural networks840), historical time clip storage860, live video stream presenter862(in the form of a server) and operator/manager web interface864.

Vehicle820is similar to vehicle520described above. Vehicle820may be in the form of a tractor, semi-trailer, cement mixer, truck, harvester, sprayer, construction vehicle, drone, plane, helicopter and the like.

Video cameras822are similar to video cameras522described above. Video cameras822-1and822-2may face in a sideways direction. Video camera822-3may face in a forward direction, either at an angle or parallel to the longitudinal axis of vehicle820. Video camera822-4faces in a rearward direction, at a rearward angle or parallel to the longitudinal axis of vehicle820.

Computers824are provided for each of cameras822. Computers824received image frames from their respective cameras and output image frames streams. In the example illustrated, computers824output, directly or indirectly, three streams: a first stream which is transmitted to a neural network840, a second stream which is transmitted to an up loader826and a third stream which is transmitted to a web RTC server828. In the example illustrated, computers824may further submit ROS frames to the smart screen831for display. Such computers824may be part of a multimedia framework such as G streamer.

Up loaders826receive video clips (a consecutive series of image frames) and wirelessly communicate such video clips to the video clip storage860located in the cloud. Such video clips870are then made available to an operator using web interface864.

Web RTC servers828, which may be in the form of Janus Gateways, wirelessly communicate the stream of image frames to the live video stream presenter862on the cloud. The live video stream presenter862may be in the form of a TURN server. The live video stream presenter862provides live streams872to an operator/manager or other person via web interface864.

As shown byFIG.4, cameras822-2and822-4as well as their associated computers824-2and824-4share up loader862-2and web RTC server828-2. Likewise, cameras822-1and822-3as well as their associated computers824-1and824-3share up loader862-1and web RTC server828-1. Image frames streams are provided to a neural network840on the vehicle820and to a smart screen831on the vehicle820. At the same time, image frame streams, live streams and video clips, are transmitted to a server862and storage860for availability on web interface864.

FIG.5illustrates an example image streaming system900comprising vehicle920in the form of a tractor having multiple cameras922. Although illustrated as a tractor, the vehicle920may alternatively comprise other forms such as a truck, a passenger automobile, a harvester, construction equipment, or other forms of agricultural, construction, commercial or other vehicles that may employ one or more cameras that offer image frame streaming. In the example illustrated, cameras922are situated on a roof of a cab of the tractor. In other implementations, camera922may be provided at other alternative or additional locations on vehicle920.

FIG.6is a schematic vehicle920and its cameras922-1,922-2,922-3,922-4, and922-5. Cameras922-1,922-2,922-3,922-4and922-5comprise two-dimensional cameras, whereas cameras922-6and922-7comprise three-dimensional or stereo cameras. Cameras922-1,922-2and922-6have fields of view facing a front of vehicle920. Cameras922-3,922-4and922-7have fields of view facing a rear of vehicle920. Camera922-5has a field-of-view encompassing the power takeoff located at the lower rear end of vehicle920. Each of such cameras provide image signals that may be used for automated or remote control of vehicle920, historical video storage for subsequent analytics, and image frame streaming for live viewing by an operator.

As further shown byFIG.6, each of cameras922is associated with a computer930(processor and non-transitory computer-readable medium) having an identifier such as 192.168.1.105. In the example illustrated, the computers are provided with the cameras which are provided in the roof of the vehicle920. In some implementations, the computers930comprise vehicle servers (labeled S3-S8inFIG.6).

Vehicle920further comprises vehicle servers926and928. Such servers serve as web real time communication (RTC) servers to communicate with remote cloud-resources for vehicle controls communications. As will be described hereafter, in some implementations, such servers may comprise a video uploader which is part of a historical time clip storage system, and a Janus Gateway server.

FIG.7is a diagram illustrating portions of system900.FIG.9illustrates an image frame streaming system associated with camera922-6. As shown byFIG.7, vehicle920additionally comprises a second server S2and additional computers (computer modules S4and S6-S8) schematically illustrated inFIG.6. Each of the additional computers (S4and S6-S8) and the additional server S2may function in the manner described below for the additional cameras622. As a result, system900provides concurrent image frame streaming from multiple cameras on the vehicle to multiple consumers.

As shown byFIG.7, computer930carries out several processes (schematically represented by gears). Computer930collects camera image frames from camera922-6and publishes them to rostopic (process934). As indicated by arrow936, a first stream of Robot Operating System (ROS) frames are created and transmitted to a first consumer in the form of a neural network940(machine learning). Such machine learning may be utilized as part of neural network training or as part of use of a neural network to carry out analytical operations. For example, neural network940may utilize the received ROS frames to identify crop, orchard or vineyard rows, to center a vehicle between such rows, to identify obstructions or obstacles and the like.

As indicated by arrow942, computer930further carries out a process938which receives a second stream of ROS frames and republishes a second stream of ROS frames which are further republished into frames that are compatible with a multimedia framework, such as a G streamer pipeline-based multimedia framework. As indicated by arrow944, a third stream of ROS frames are transmitted to a second consumer in the form of smart screen948. Smart screen948comprises a display to present the received ROS frames. In some implementations, smart screen948and stream944may be omitted.

As indicated by arrow950, computer930carries out process952to subscribe or receive the second stream of ROS frames to run the G streamer pipelines. As indicated by arrow954, a first stream or pipeline from the multimedia framework (Gstreamer in the example) is transmitted as videos or video clips to a video uploader956which is part of a historical time clip storage system. As indicated by arrow958, uploader956uploads the videos to a cloud-based storage960. As indicated by arrow962, such video clips963stored in storage960may be retrieved by a client/operator/manager browser964having a display for presentation to a person, such as an operator or manager. Although the video clips in the example are five minutes in length, in other implementations, the video clips may have other durations. The browser964may be part of a smart phone, a tablet computing device, a laptop computing device, a desktop computing device monitor, display screen or the like.

As indicated by arrow966, a stream of image frames is further sent to the web RTC server968. The web RTC server968cooperates with a cloud-based TURN server970and display964to provide sub second latency live meteor camera streams972which are presented on display964.

FIG.8is a block diagram schematically illustrating an example live streaming architecture974of system900. Architecture974comprises a web real time communication server968and TURN server970. Server968may be in the form of a Janus Gateway, an open-source service. The web RTC server serves as a stun server980and a signaling server982. The stunned server enables clients/operators to find out their public IP addresses, NAT type and Internet facing port associated by the NAT device with a particular local port. This information may be used to set up a UDP/TCP communication session between the peers, two endpoints, for peer-to-peer connection. In the example illustrated, the two peers are the vehicle/tractor920(vehicle computer) and the browser964.

The signaling server is a server that manages a connection between peers. The signaling server does not deal with media traffic itself but takes care of signaling. Such signaling includes enabling one user to find another in the network, negotiating the connection itself, resetting the connection if needed and closing the connection down.

The Traversal Using Relays around NAT (TURN) server970is a protocol that assists in the traversal network address translator's (NAT) or firewalls984for WebRTC applications. The TURN server970facilitates ascending and transmission of data by clients through an intermediary server. In some implementations, the TURN protocol may be an extension to the STUN server.

In the example illustrated, a web RTC framework is built to serve as a powerful tool to infuse real time communication (RTC) capabilities into browsers, such browser964and mobile applications. The web RTC framework is implemented for live video streaming using the Janus Gateway968. As noted above, Janus Gateway968serves as both the stun server and a signaling server, exchanging session description protocols (SDPs) in establishing connections between peers. Session description protocols are a set of rules and define how multimedia sessions can be set up to allow all endpoints (peers) to effectively participate in a session. A session may comprise a set of communication endpoints along with a series of interactions amongst them. A session may comprise a json object which contains information about a peer, such as public IP address, codec and the like, to facilitate peer-to-peer connections. The Janus Gateway server is configured on the vehicle/tractor920(the vehicle computer). The Traversal Using Relays around NAT (network address translation) (TURN) server is built or configured in a server in the cloud (such as Amazon cloud (AWS Cloud)). In some implementations, the TURN server may be established or created on a pool of computer instances which are mapped to a network load balancer.

On each of the computers associated with one of cameras922, G streamer pipelines are executed. Payload of a pipeline is a video codec (VP9). The sink of each pipeline is udpsink (a network sink that sends UDP packets to the network). It can be combined with RTP payloaders to implement RTP streaming to transmit streams to VEHICLE COMPUTER private ip using respective ports. The vehicle side of the WebRTC Framework968listens to the ports to determine which computers930(Computer's) are transmitting streams to process the same to a connected peer. When an offer from the client browser964and an answer from the tractor NUC1 are exchanged, signaling server982establishes the shortest possible connection between both of the peers (browser964and tractor/vehicle920) to facilitate communication.

In some implementations, a pool of computer instances or TURN servers may be prepared. Load-balancing may be carried out by using an available one of the TURN servers. In some implementations, the TURN server listener group may be auto scaled. In some implementations, the TURN server970( ) may be monitored. If the TURN server970dies, auto healing may be performed. In some implementations, if the vehicle/tractor920Internet speed is less than 10 Mb per second, streaming may be discontinued or restricted to a portion of cameras922. In some implementations, the transmit streams may be based upon Wi-Fi speed of the tractor/vehicle920, providing adaptive bit rate. In some implementations, a user may log into the system for such live streaming. In some implementations, the TURN server970may be configured as a service (server last). In such implementations, the TURN server may be secured against public use.