Synchronization controller unit for sensor node

A vehicular synchronization system includes an image sensor of a vehicle. The image sensor captures image data for an image sensor field of view. The vehicular synchronization system also includes a light sensor having a radial view. The light sensor captures light data from around the vehicle in a full rotation. The vehicular synchronization system includes a synchronization control unit having a data receiving module, a trigger time determination module, a timer module, and a sensor trigger module. The data receiving module receives a data packet from the light sensor. The trigger time determination module determines a trigger time when a desired light sensor orientation overlaps with the image sensor field of view. The timer module sets a timer to elapse at the trigger time. The sensor trigger module controls the image sensor to capture image data at the trigger time when the timer elapses.

BACKGROUND

Increasingly, vehicles are integrating real time sensor data from multiple sensors with operational control of the vehicle. The sensor data takes into account objects (e.g., roadways, obstacles, other vehicles) that may be faced by the vehicle during vehicle operation in real-time. However, the disparate sensors act independently and therefore, produce uncorrelated data streams. For example, a first sensor and a second sensor may have different timing mechanisms that do not account for inherent latencies or delays. Accordingly, the data streams from the first sensor and the second sensor cannot be chronologically synched. Consequently, despite having a number of sensors, it might not be possible to determine a complete driving scene. This limits how well the sensor data is integrated with control systems of the vehicle.

BRIEF DESCRIPTION

According to one aspect, a vehicular synchronization system includes an image sensor of a vehicle. The image sensor captures image data for an image sensor field of view. The vehicular synchronization system also includes a light sensor having a radial view. The light sensor captures light data from around the vehicle in a full rotation of the light sensor. The vehicular synchronization system further includes a synchronization control unit having a data receiving module, a trigger time determination module, a timer module, and a sensor trigger module. The data receiving module receives a data packet from the light sensor. The data packet identifies a current orientation of the light sensor. The trigger time determination module determines a trigger time when a desired light sensor orientation overlaps with the image sensor field of view based on the current orientation of the light sensor. The timer module sets a timer to elapse at the trigger time. The sensor trigger module controls the image sensor to capture image data at the trigger time when the timer elapses.

According to another aspect, a computer-implemented method for synchronizing sensor nodes is described. The computer-implemented method includes receiving a data packet from a light sensor having a radial view. The light sensor captures light data from around a vehicle in a full rotation. The data packet identifies a current orientation of the light sensor. The computer-implemented method also includes determining a trigger time when a desired light sensor orientation overlaps with an image sensor field of view of an image sensor based on the current orientation of the light sensor. The computer-implemented method further includes setting a timer to elapse at the trigger time. The computer-implemented method includes controlling the image sensor to capture image data at the trigger time. The computer-implemented method further includes updating the timer for an upcoming trigger time when a next desired light sensor orientation overlaps with the image sensor field of view.

According to another aspect, a vehicular synchronization system includes an image sensor. The image sensor captures image data in an image sensor field of view. The vehicular synchronization system also includes a light sensor having a radial view. The light sensor captures light data from around the vehicle in a full rotation. The vehicular synchronization system further includes a synchronization control unit having a data receiving module, a trigger time determination module, a timer module, and a sensor trigger module. The data receiving module receives a data packet from the light sensor. The data packet identifies a current orientation of the light sensor. The trigger time determination module determines a trigger time when a desired light sensor orientation overlaps with the image sensor field of view based on the current orientation of the light sensor. The timer module sets a timer to elapse at the trigger time. The sensor trigger module controls the image sensor to capture image data at the trigger time when the timer elapses. The vehicular synchronization system further includes a robot operating system having a light sensor node and at least one image sensor node. The light sensor node receives the captured light data. The at least one image sensor node receives the captured image data. Synchronized sensor data is generated by combining the captured image data with the captured light data based on the trigger time.

DETAILED DESCRIPTION

The following includes definitions of selected terms employed herein. The definitions include various examples and/or forms of components that fall within the scope of a term and that can be used for implementation. The examples are not intended to be limiting.

A “bus”, as used herein, refers to an interconnected architecture that is operably connected to other computer components inside a computer or between computers. The bus can transfer data between the computer components. The bus can be a memory bus, a memory controller, a peripheral bus, an external bus, a crossbar switch, and/or a local bus, among others. The bus can also be a vehicle bus that interconnects components inside a vehicle using protocols such as Media Oriented Systems Transport (MOST), Controller Area network (CAN), Local Interconnect Network (LIN), among others.

A “disk”, as used herein can be, for example, a magnetic disk drive, a solid state disk drive, a floppy disk drive, a tape drive, a Zip drive, a flash memory card, and/or a memory stick. Furthermore, the disk can be a CD-ROM (compact disk ROM), a CD recordable drive (CD-R drive), a CD rewritable drive (CD-RW drive), and/or a digital video ROM drive (DVD ROM). The disk can store an operating system that controls or allocates resources of a computing device.

A “memory”, as used herein can include volatile memory and/or non-volatile memory. Non-volatile memory can include, for example, ROM (read only memory), PROM (programmable read only memory), EPROM (erasable PROM), and EEPROM (electrically erasable PROM). Volatile memory can include, for example, RAM (random access memory), synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and direct RAM bus RAM (DRRAM). The memory can store an operating system that controls or allocates resources of a computing device.

A “module”, as used herein, includes, but is not limited to, non-transitory computer readable medium that stores instructions, instructions in execution on a machine, hardware, firmware, software in execution on a machine, and/or combinations of each to perform a function(s) or an action(s), and/or to cause a function or action from another module, method, and/or system. A module may also include logic, a software controlled microprocessor, a discrete logic circuit, an analog circuit, a digital circuit, a programmed logic device, a memory device containing executing instructions, logic gates, a combination of gates, and/or other circuit components. Multiple modules may be combined into one module and single modules may be distributed among multiple modules.

An “operable connection”, or a connection by which entities are “operably connected”, is one in which signals, physical communications, and/or logical communications can be sent and/or received. An operable connection can include a wireless interface, a physical interface, a data interface and/or an electrical interface.

A “processor”, as used herein, processes signals and performs general computing and arithmetic functions. Signals processed by the processor can include digital signals, data signals, computer instructions, processor instructions, messages, a bit, a bit stream, or other means that can be received, transmitted and/or detected. Generally, the processor can be a variety of various processors including multiple single and multicore processors and co-processors and other multiple single and multicore processor and co-processor architectures. The processor can include various modules to execute various functions.

A “vehicle”, as used herein, refers to any moving vehicle that is capable of carrying one or more vehicle occupants and is powered by any form of energy. The term “vehicle” includes, but is not limited to: cars, trucks, vans, minivans, SUVs, motorcycles, scooters, boats, go-karts, amusement ride cars, rail transport, personal watercraft, and aircraft. In some cases, a motor vehicle includes one or more engines. Further, the term “vehicle” can refer to an electric vehicle (EV) that is capable of carrying one or more vehicle occupants and is powered entirely or partially by one or more electric motors powered by an electric battery. The EV can include battery electric vehicles (BEV) and plug-in hybrid electric vehicles (PHEV). The term “vehicle” can also refer to an autonomous vehicle and/or self-driving vehicle powered by any form of energy. The autonomous vehicle may or may not carry one or more vehicle occupants. Further, the term “vehicle” can include vehicles that are automated or non-automated with pre-determined paths or free-moving vehicles.

A “vehicle occupant,” as used herein can include, but is not limited to, one or more biological beings located in the vehicle. The vehicle occupant can be a driver or a passenger of the vehicle. The vehicle occupant can be a human (e.g., an adult, a child, an infant) or an animal (e.g., a pet, a dog, a cat).

A “vehicle system,” as used herein can include, but is not limited to, any automatic or manual systems that can be used to enhance the vehicle, driving, and/or safety. Exemplary vehicle systems include, but are not limited to: an electronic stability control system, an anti-lock brake system, a brake assist system, an automatic brake prefill system, a low speed follow system, a cruise control system, a collision warning system, a collision mitigation braking system, an auto cruise control system, a lane departure warning system, a blind spot indicator system, a lane keep assist system, a navigation system, a transmission system, brake pedal systems, an electronic power steering system, visual devices (e.g., camera systems, proximity sensor systems), a climate control system, an electronic pretensioning system, a monitoring system, a passenger detection system, a vehicle suspension system, a vehicle seat configuration system, a vehicle cabin lighting system, an audio system, a sensory system, among others.

A “value” and “level”, as used herein can include, but is not limited to, a numerical or other kind of value or level such as a percentage, a non-numerical value, a discrete state, a discrete value, a continuous value, among others. The term “value of X” or “level of X” as used throughout this detailed description and in the claims refers to any numerical or other kind of value for distinguishing between two or more states of X. For example, in some cases, the value or level of X may be given as a percentage between 0% and 100%. In other cases, the value or level of X could be a value in the range between 1 and 10. In still other cases, the value or level of X may not be a numerical value, but could be associated with a given discrete state, such as “not X”, “slightly x”, “x”, “very x” and “extremely x”.

Referring now to the drawings, the showings are for purposes of illustrating one or more exemplary embodiments and not for purposes of limiting same.FIG. 1is a schematic view of an exemplary operating environment100for implementing systems and methods for synchronizing sensors with a synchronization controller unit according to an exemplary embodiment. The components of the operating environment100, as well as the components of other systems, hardware architectures, and software architectures discussed herein, can be combined, omitted, or organized into different architectures for various embodiments.

In the illustrated embodiment ofFIG. 1, the operating environment100includes a vehicle computing device (VCD)102with provisions for processing, communicating, and interacting with various components of a vehicle and other components of the operating environment100. In one embodiment, the VCD102can be implemented with a host vehicle202(FIG. 2), for example, as part of a telematics unit, a head unit, a navigation unit, an infotainment unit, an electronic control unit, among others. In other embodiments, the components and functions of the VCD102can be implemented remotely from the host vehicle202, for example, with a portable device (not shown) or another device connected via a network (e.g., a network130).

Generally, the VCD102includes a processor104, a memory106, a disk108, and a synchronization control unit110, and an input/output (I/O) interface122, which are each operably connected for computer communication via a bus124and/or other wired and wireless technologies. The I/O interface122provides software and hardware to facilitate data input and output between the components of the VCD102and other components, networks, and data sources, which will be described herein. The synchronization control unit110includes a data receiving module112, a trigger time determination module114, a timer module116, a sensor trigger module118, and an update module120, each suitable for controlling vehicle sensors126and vehicle systems128of the host vehicle202.

The synchronization control unit110may be configured to operably control a plurality of components of the host vehicle202. The synchronization control unit110may additionally provide one or more commands to one or more control units (not shown) of the host vehicle202, including, but not limited to a sensor control unit, an engine control unit, a braking control unit, a transmission control unit, a steering control unit, and the like to control the host vehicle202to trigger sensors, brake, be autonomously driven, etc. In one or more embodiments, the synchronization control unit110may include a microprocessor, one or more application-specific integrated circuit(s) (ASIC), or other similar devices. The synchronization control unit110may also communicate with the memory106to execute the one or more applications, operating systems, vehicle systems128, subsystems thereof, user interfaces, and the like.

The VCD102is operably connected for computer communication (e.g., via the I/O interface122and/or the bus124) to one or more of the vehicle sensors126. The vehicle sensors126may be associated with vehicle systems128and other vehicle sensors associated with the host vehicle202. For example, the vehicle sensors126can include, but are not limited to, image sensors, such as cameras, optical sensors, radio sensors, etc. mounted to the interior or exterior of the host vehicle202and light sensors, such as light detection and ranging (LiDAR) sensors, radar, laser sensors etc. mounted to the exterior of the host vehicle202. The vehicle sensors126associated with the vehicle systems128may include vehicle speed sensors, accelerator pedal sensors, brake sensors, throttle position sensors, wheel sensors, anti-lock brake sensors, camshaft sensors, among others. Other vehicle sensors can include, but are not limited to, sensors external to the vehicle (accessed, for example, via the network130), for example, external cameras, radar and laser sensors on other vehicles in a vehicle-to-vehicle network, street cameras, surveillance cameras, among others.

The vehicle sensors126are operable to capture data regarding the vehicle, the vehicle environment, and/or the vehicle systems128. The captured data may take the form of a generated data signal indicating a value corresponding to measured data. These data signals can be converted into other data formats (e.g., numerical) and/or used by the vehicle systems128and/or the VCD102to generate other data metrics, levels, and values.

FIG. 3illustrates example vehicle sensors126that may be mounted to the host vehicle202according to an exemplary embodiment. It is understood that the vehicle sensors126shown inFIG. 3are exemplary in nature and other vehicle sensors can be implemented with the systems and methods discussed herein. In the embodiment shown inFIG. 3, the vehicle sensors126include camera system302. The camera system302may be comprised of one or more image sensors208. In the embodiment shown inFIG. 3, one or more image sensors208of the camera system302includes a first camera304, a second camera306, and a third camera308. More or fewer image sensors may be used and/or different types of image sensors may be used. The one or more image sensors208may be mounted at different points around the exterior of the host vehicle202.

With regard toFIG. 2, the one or more image sensors208may be positioned in a direction to capture the surrounding environment of the host vehicle202. In an exemplary embodiment, the surrounding environment of the host vehicle202may be defined as a predetermined area located around (front/sides/behind) the host vehicle202(e.g., road environment in front, sides, and/or behind of the host vehicle202) that may be included within the vehicle's travel path. The one or more image sensors208may be disposed at external front and/or side portions of the host vehicle202, including, but not limited to different portions of the vehicle bumper, vehicle front lighting units, vehicle fenders, and the windshield. The one or more image sensors208may be positioned on a planar sweep pedestal (not shown) that allows the one or more image sensors208to be oscillated to capture images of the external environment of the host vehicle202at various angles. Additionally, the one or more image sensors208may be disposed at internal portions of the host vehicle202including the vehicle dashboard (e.g., dash mounted camera), rear side of a vehicle rear view mirror, etc.

Returning toFIG. 3, the one or more image sensors208may include the first camera304, the second camera306, and the third camera308of the camera system302being placed at different points around the host vehicle202to image different fields of view. In another embodiment, the first camera304, the second camera306, and the third camera308may have at least partially overlapping fields of view to achieve a stereoscopic image that creates the impression of depth and solidarity, for example, to enable a three-dimensional effect. While three cameras are described, more or fewer cameras may be used by the camera system302.

The vehicle sensors126may also include a LiDAR system310. The LiDAR system310may comprise one or more light sensing transceivers204that include one or more planar sweep lasers that include respective three-dimensional LiDAR sensors that may be configured to oscillate or rotate. The one or more light sensing transceivers204emit one or more laser beams of ultraviolet, visible, or near infrared light toward the surrounding environment of the host vehicle202. The one or more light sensing transceivers204may be configured to receive one or more reflected laser waves (e.g., signals) that are reflected off one or more objects included within the surrounding environment of the host vehicle202. In other words, upon transmitting the one or more laser beams to the surrounding environment of the host vehicle202, the one or more laser beams may be reflected as laser waves by one or more traffic related objects (e.g., motor vehicles, pedestrians, trees, guardrails, etc.) that are located within the surrounding environment of the host vehicle202and is received back at the one or more light sensing transceivers204.

The vehicle systems128include and/or are operably connected for computer communication to various vehicle sensors126. The vehicle sensors126provide and/or sense information associated with the host vehicle202, the vehicle environment, and/or the vehicle systems128. The vehicle systems128can include, but are not limited to, any automatic or manual systems that can be used to enhance the vehicle, driving, and/or safety. In the embodiment shown inFIG. 3, the vehicle systems128can include a navigation system312, a robot operating system314, and an infotainment system316.

The navigation system312may be primarily used to identify a location of the host vehicle202and calculate directions to a destination. The navigation system312includes a global positioning system (GPS) receiver318to determine geolocation and time information. In addition to using this information to identify locations and calculate directions, the geolocation and time information may be used by other vehicle systems128and the vehicle sensors126. The time information may be based on a time stamp signal from a reference clock320of the GPS receiver318. The reference clock320outputs the time stamp signal and repeats the time stamp signal at a known interval. For example, the reference clock320may output a time stamp signal having a width of less than one second and a sharply rising or abruptly falling edge that accurately repeats once per second.

The robot operating system314acts as a data platform for the vehicle systems128. In particular, the robot operating system314manages nodes that communicate between the different components of the operating environment100. For example, the vehicle sensors126may have corresponding sensor nodes in the robot operating system314that receive and transmit sensor data from the vehicle sensors126, as will be described in more detail with respect toFIG. 4. Likewise, the synchronization control unit110may have synchronization control nodes that receive and transmit data from the synchronization control unit110, which will also be described in more detail with respect toFIG. 4. The robot operating system314may be the Robot Operating System that is the subject of an open source project created by Willow Garage and Stanford Artificial Intelligence (Al) Laboratory or another data platform having one or a set of utilities that receive, manage, create, and/or send nodes may be used for the robot operating system314. The nodes may be components or programs associated with the vehicle sensors126and/or vehicle systems128. The nodes send and receive messages regarding the associated vehicle sensors126and/or vehicle systems128in the operating environment100, as will be discussed in more detail below.

Further, the vehicle systems128can include an infotainment system316. The infotainment system316may include an in-vehicle display322. As will be discussed in more detail herein, the synchronization control unit110causes the synchronization at least two of the vehicle sensors126using the robot operating system314. The synchronized sensor data from the synchronized vehicle sensors may be displayed on the in-vehicle display322. The infotainment system316may additionally cue a vehicle occupant that the synchronized sensor data is available through an audio or visual alerts.

Returning toFIG. 1, the VCD102is also operatively connected for computer communication to the network130and a sensor database132. It is understood that the connection from the I/O interface122to the network130and the sensor database132can be facilitated in various ways. For example, through a network connection (e.g., wired or wireless), a cellular data network from a portable device (not shown) or a vehicle to vehicle ad-hoc network (not shown), an in-vehicle network (not shown), among others, or any combination of thereof.

The network130is, for example, a data network, the Internet, a wide area network or a local area network. The network130serves as a communication medium to various remote devices (e.g., databases, web servers, remote servers, application servers, intermediary servers, client machines, other portable devices). It is understood that in some embodiments, the sensor database132can be included in the network130, accessed by the VCD102through the network130, and/or the network130can access the sensor database132. Thus, in some embodiments, the VCD102can obtain data from the sensor database132via the network130.

The sensor database132can include sensor information from external sensors, such as traffic cameras, in-road sensors, and the sensors of proximal vehicles to the host vehicle202, such as cameras. The sensor database132may identify objects in the area of the host vehicle202in real-time using the external sensors. The sensor database132may include historical information about permanent and semi-permanent objects such as geological objects (e.g., bodies of water, landforms, perennial plants, etc.), man-made objects (e.g., bridges, traffic signals, buildings, etc.), and transitory objects (e.g., people in a crosswalk, dogs chasing cars, proximal vehicle, etc.). In some embodiments, the sensor database132can be updated with object information on a continual or periodic basis. The sensor database132can aggregate data from the VCD102, the vehicle sensors126, and the vehicle systems128.

It is understood that the sensor database132can be located remotely from the VCD102and accessed, for example, by the network130. In some embodiments, the sensor database132could be located on-board the host vehicle202, at for example, the memory106and/or the disk108. Further, in some embodiments, the sensor database132could be located on an external memory or a disk (not shown) integrated. In other embodiments, the sensor database132could be distributed in one or more locations.

The system shown inFIG. 1will now be described in operation according to an exemplary embodiment over a time200, shown inFIG. 2. As discussed above, and as shown in detail inFIG. 2, the system includes a host vehicle202having one or more light sensing transceivers204. The one or more light sensing transceivers204are sensitive to electromagnetic radiation in radial view206. A light sensing transceivers204rotates about its central axis. Accordingly, the one or more light sensing transceivers204can capture data from 360 degrees around the host vehicle202in time200. In this manner, the one or more light sensing transceivers204may be continuously sensitive to electromagnetic radiation for a rotating radial view206.

The one or more image sensors208have at least one field of view corresponding to a field of view210. The one or more image sensors208capture data from the environment of the host vehicle202as one or more images. For example, as described above, the image sensors may be cameras that capture image frames. The one or more image sensors208may have a stationary field of view210or alternatively the field of view210may pan at least partially around the host vehicle202. In some embodiments, the field of view210may be discretized over areas corresponding to the one or more image sensors208either individually or in combination. For example, each camera may have a corresponding field of view or multiple cameras may have a combined field of view.

As the one or more light sensing transceivers204rotate in time200, the radial view206may periodically overlap with the field of view210in a synchronization area212. The synchronization area212is defined by the overlap of at least a portion of the radial view with at least a portion of the field of view210. The synchronization area212may be bounded by either a region of the radial view206or the field of view210. Alternatively, the radial view206and the field of view210may be aligned on at least one side, such that the at least one side bounds the synchronization area212from an un-sensed region of the environment of the host vehicle202.

Objects, such as, at least a portion of a proximal vehicle214may be in the synchronization area212. Other objects may be sensed by one sensor but not another. For example, the street light216and/or a tree218may be in the field of view210of the one or more image sensors208but not in the synchronization area212. Instead, the street light216may be the synchronization area212at a later point in time200. Likewise, the tree218may have been in the synchronization area212at a previous point in time200.

The host vehicle202includes the synchronization control unit110, described above with respect toFIG. 1, which synchronizes the one or more light sensing transceivers204and the one or more image sensors208. The data receiving module112receives data from a number of sources. For example, the data receiving module112may receive a time-based schedule for triggering the one or more image sensors208. In another embodiment, the data receiving module112may receive an orientation-based schedule for triggering the one or more image sensors208, based on a desired orientation of the one or more light sensing transceivers204. Additionally, the synchronization control unit110may receive a data packet from the one or more light sensing transceivers204. The data packet may include a current orientation of the one or more light sensing transceivers204.

In some embodiments, the data packet includes a light sensor time stamp. The light sensor time stamp may be based on a timing reference signal from a reference clock320. The reference clock320may be a component integrated with the GPS receiver318of the navigation system312. In another embodiment, a timing reference signal may integrated with the one or more light sensing transceivers204. In some embodiments, the timing reference signal is based on a standardized on time. For example, the timing reference signals may be based on a global positioning system (GPS).

The trigger time determination module114determines a trigger time220when the radial view206of the one or more light sensing transceivers204coincides with a desired light sensor orientation. At the desired light sensor orientation, the radial view206overlaps with the field of view210creating the synchronization area212. In some embodiments, the trigger time determination module114may determine the trigger time220based on the time-based schedule or the orientation-based schedule. In another embodiment, the trigger time determination module114may calculate the trigger time220based on information from the data packet such as the current orientation of the one or more light sensing transceivers204and a light sensor time stamp. For example, based on a time stamp, orientation, and rate of rotation, an estimated time of when to trigger a camera may be calculated.

The timer module116sets a timer to elapse at the trigger time220. When the timer elapses, the sensor trigger module118controls at least one image sensor of the one or more image sensors208to activate such that the at least one image sensor captures image data in the synchronization area212. Suppose that the rotation of the one or more light sensing transceivers204is counter clockwise in time200. In some embodiments, the trigger time220may be set to incorporate latencies so that the one or more image sensors208activate at an activation time222.

The radial view206has a radial longitudinal axis, such that the radial view206is symmetrical about the radial longitudinal axis. The field of view210has a camera central axis, such that the field of view210is symmetrical about the camera longitudinal axis. In some embodiments, the activation time222may be when the one or more light sensing transceivers204is orientated such that the radial longitudinal axis overlaps with the camera central axis.

The update module120updates the timer for an upcoming trigger time when a next desired light sensor orientation of the one or more light sensing transceivers204overlaps with the image sensor field of view. The upcoming trigger time may be based on the time-based schedule or the orientation-based schedule. In another embodiment, the trigger time determination module114may calculate the next trigger time based on the time-based schedule, the orientation-based schedule, and/or information from a data packet of the one or more light sensing transceivers204. The data packet may be a previous data packet or an updated data packet from the one or more light sensing transceivers204.

In some embodiments, the synchronization control unit110utilizes the robot operating system314to synchronize the one or more light sensing transceivers204and the one or more image sensors208.FIG. 4is a schematic diagram of a vehicle implementing a system for synchronizing the sensors with sensor nodes according to an exemplary embodiment. For example, the first camera304, the second camera306, and the third camera308may communicate through a first channel402and the LiDAR system310may communicate through a second channel404. Although illustrated as different channels, the first channel402and the second channel404may be the same channel. In another embodiment, the first channel may be a hub device and the second channel404may be an ethernet switch. In another embodiment, the sensor data may be from the sensor database132and the channel may be the network130.

The first camera304, the second camera306, and the third camera308may communicate through the first channel402according to a first driver408. The LiDAR system310may communicate through the second channel404according to a second driver410. The robot operating system314may operate subject to or as a part of a vehicle system128or operate in conjunction with the processor104.

The robot operating system314has sensor nodes corresponding to the sensors. For example, the first camera304, the second camera306, and the third camera308correspond to a first camera node412, a second camera node414, and a third camera node416, respectively. Likewise, the LiDAR system310corresponds to a LiDAR node418. The sensor nodes are capable of sending and receiving messages with other components of the operating environment100including the vehicle sensors126and the vehicle systems128. For example, the first camera node may receive image data, such as image frames captured by the one or more image sensors208.

The robot operating system314also includes synchronization control nodes. The synchronization control nodes correspond to sensor nodes for sensors that do not have access to a timing reference signal. In this example, suppose that the first camera304, the second camera306, and the third camera308do not have access to a timing reference signal that issues a time stamp based on a standardized time. Accordingly, the first camera node412, the second camera node414, and the third camera node416correspond a first synchronization control node420, a second synchronization control node422, and a third synchronization control node424, respectively. The synchronization control nodes are capable of sending and receiving messages from the synchronization control unit110with other components of the operating environment100including other nodes, the vehicle sensors126, and/or the vehicle systems128. In one embodiment, the synchronization control nodes include trigger time data, such as trigger time220and/or activation time222determined by the trigger time determination module114of the synchronization control unit110.

The sensor nodes and the synchronization control nodes communicate with time stamp nodes. The time stamp nodes correlate sensor data of the sensor nodes with the trigger time data of the synchronization control nodes. For example, the first time stamp node426correlates image data from the first camera node412with trigger time data from the first synchronization control node420. Accordingly, the first time stamp node426can time stamp the image data based on when the first camera304was triggered by the synchronization control unit110. By correlating the first camera node412with the first synchronization control node420, the first time stamp node426avoids latency issues with receiving the image data. For example, if it takes the first camera304one second to capture and deliver image data, a one second delay is incurred by the robot operating system314.

Suppose that the LiDAR system310operates using the reference clock320of the navigation system312or has an internal reference clock. Light data received from the LiDAR system310may not be subject to the one second delay. Accordingly, when attempting to associate with the image data with the light data the light data would be from one second earlier than the image data. In this example, the light data from the one or more light sensing transceivers204would illustrate the proximal vehicle214but the image data of the one or more image sensors208would illustrate the street light216. Accordingly, the first time stamp node426, the second time stamp node428, and the third time stamp node430correlate the image data with the trigger time data. Thereby, avoiding propagating delay errors inherent in processing the sensor data.

The correlated data is received by the nodal camera synch data432. The nodal camera synch data432may use the data from the first time stamp node426, the second time stamp node428, and the third time stamp node430to create a stereoscopic image based on the image data of the first camera304, the second camera306, and the third camera308being triggered at the trigger time. The nodal camera synch data432can be stored in the memory106or displayed on the in-vehicle display322.

The correlated data from the first time stamp node426, the second time stamp node428, and the third time stamp node430can also be combined with the light data of the LiDAR node418to generate synched sensor data434. The synched sensor data434can be stored in the memory106or displayed on the in-vehicle display322. Alternatively, the vehicle systems128may use the synched sensor data434for operational functionality, such as autonomous driving.

FIG. 5is a process flow diagram of a method for synchronizing sensors with a synchronization controller unit according to an exemplary embodiment.FIG. 5will be described with reference toFIGS. 1, 2, and 4. It is understood that the illustrative examples discussed herein are exemplary in nature and that other vehicle sensors126, vehicle systems128, and vehicle control functions can be implemented.

At block502, the method includes receiving a data packet from a light sensor having a radial view. The light sensor may be a light sensor transceiver of the one or more light sensing transceivers204such a rotating LiDAR sensor of a LiDAR system310. The light sensor is configured to capture light data from around a vehicle. For example, the light sensor may rotate 360 degrees around the host vehicle202. The light sensor generates a data packet that identifies a current orientation of the light sensor. The data packet may also include a light sensor time stamp based on a timing reference signal. The timing reference signal may be generated by the light sensor itself or be received from a reference clock320.

In some embodiments, the light sensor may periodically generate updated data packets. For example, the light sensor may generate a data packet according to a time interval (e.g., every 46 microseconds). Alternatively, the light sensor may generate a data packet according to an angular interval (e.g., every 5 degrees of the 360 degrees traversed by the light sensor).

At block504, the method includes determining a trigger time when a desired light sensor orientation of the light sensor overlaps with an image sensor field of view of an image sensor. The trigger time may be based on the data packet and following updated data packets. The trigger time may be defined as the time when the radial view206first overlaps with the field of view210defining the boundary of the synchronization area212.

At block506, the method includes setting a timer to elapse at the trigger time. The timer may be a component of synchronization control unit110or as a part of the vehicle sensors126or the vehicle systems128

At block508, the method includes controlling the image sensor to capture image data at the trigger time. In another embodiment, the image sensor may be actuated at the trigger time but due to inherent latencies, capture image data at the activation time. However, the image sensor may not have access to a timing mechanism, and accordingly, the captured image data is not associated with a time stamp.

At block510, the method includes updating the timer for an upcoming trigger time when a next desired light sensor orientation overlaps with the image sensor field of view. Accordingly, the synchronization control unit110prepares to capture updated image data at the upcoming trigger time. The upcoming trigger time may be based on the updated data packets of the light sensor.

At block512, the method includes receiving the captured image data at the at least one image sensor node. The captured image data may include image data such as image frames from the image sensors.

At block514, the method includes receiving timing information for the image sensors at a synchronization control node corresponding to the at least one image sensor node. The synchronization control node may receive timing information from the synchronization control unit110. The timing information indicates the time at which the image sensor captured the image data. For example, the timing information may include trigger time220. Alternatively, the synchronization control unit110may have received data at the data receiving module112regarding a latency of image sensor. Accordingly, the timing information may include the activation time222to take into account the latency of the image sensor.

At block516, the method includes generating image synched data at a time stamp node by combining the captured image data with the timing information. The image synched data accurately indicates the time at which the image data was captured without incurring any delay in the image data being processed from the image sensor, through the first channel402according to the first driver408to the image sensor node.

At block518, the method includes generating sensor synched data by combining the image synched data with light data. In some embodiments, the light data includes the light sensor time stamp. Accordingly, the image synched data is combined with the light data by correlating the timing information with the light sensor time stamp to ensure that the image synched data was captured at the same time that light data was captured. The timing of the image sensor and the light sensor is synchronized according to the trigger time, or in some embodiments the activation time, which is the time at which radial view206of the light sensor at least partially overlaps with the field of view210of the image sensor. Because the image data and the light data are synchronized, both capture data regarding the same objects. Accordingly, different types of sensors can provide a myriad of different data regarding the objects, such as the position, speed, shape, etc. of the objects. Thus, the sensor synched data can be used by vehicle systems to facilitate the vehicle operation, such as for autonomous driving.

The examples are described with respect to an image sensor and the light sensor. These are merely exemplary in nature and other sensors may be used. For example, the time stamp nodes may combine light data from a light sensor node with timing information regarding the light sensor rather than the image sensors as described. Furthermore, it is understood that the sensors can be any type of sensor, for example, acoustic, electric, environmental, optical, imaging, light, pressure, force, thermal, temperature, proximity, among others and operate in a similar manner as the image sensor or light sensor described. For example, image data may be synched with thermal sensor data to determine if objects in the image data are emitting heat and therefore, may be alive.

It should be apparent from the foregoing description that various exemplary embodiments of the invention may be implemented in hardware. Furthermore, various exemplary embodiments may be implemented as instructions stored on a non-transitory machine-readable storage medium, such as a volatile or non-volatile memory, which may be read and executed by at least one processor to perform the operations described in detail herein. A machine-readable storage medium may include any mechanism for storing information in a form readable by a machine, such as a personal or laptop computer, a server, or other computing device. Thus, a non-transitory machine-readable storage medium excludes transitory signals but may include both volatile and non-volatile memories, including but not limited to read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, and similar storage media.

It should be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the invention. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in machine readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown. It will be appreciated that various implementations of the above-disclosed and other features and functions, or alternatives or varieties thereof, may be desirably combined into many other different systems or applications. Also that various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.