Patent Description:
The various embodiments relate generally to computer vision systems and processing and, more specifically, to techniques for roll effect correction for optical sensors.

Many vehicles today can be equipped with computer vision capabilities and techniques. Computer vision techniques for vehicles include acquiring optical data (e.g., capture images) of the environment around the vehicle using optical sensors. For example, a forward-facing optical sensor could acquire optical data of the environment in front of the vehicle. Computer vision techniques using optical data of the front environment of the vehicle can have multiple uses, including, for example, people/object detection. Such techniques have multiple applications in driver assistance and/or autonomous driving contexts, including navigational assistance, collision avoidance, driver situational awareness, autonomous vehicle operation, and/or the like.

Typically, computer vision techniques for vehicle use are trained and/or developed using optical data obtained from four-wheeled vehicles that rarely experience any appreciable rolling (e.g., sedans, coupes, sport utility vehicles, minivans, vans, trucks, and/or the like). In such optical data, the view-up orientation of the optical sensor, and correspondingly the view-up orientation of the optical data, is substantially perpendicular with the road surface. When those computer vision techniques are applied to such optical data, the detection accuracy is typically very high. However, a drawback of such techniques is that those techniques are less effective when used with vehicles (e.g., motorcycles, scooters, etc.) that can roll more appreciably than four-wheeled vehicles. When a vehicle rolls relative to the road surface, the view-up orientation of the optical sensor, and correspondingly the view-up orientation of the optical data, is no longer substantially perpendicular with the road surface. When computer vision techniques are applied to such optical data, the detection accuracy is significantly lower. With vehicles that frequently experience rolling (e.g., motorcycles), the lower accuracy can occur quite often, making such computer vision techniques less ineffective when implemented on such vehicles. <CIT> relates to a method of stabilising the image of a camera sensor using a combination of electronic and mechanical compensation. Document <CIT> discloses a method of stabilizing a payload fitted in a carrier including providing a first carrier component of the carrier, supporting a second carrier component of the carrier using the first carrier component, and supporting a third carrier component of the carrier using the second carrier component. The first carrier component is configured to permit rotation of the payload about a pitch axis. The second carrier component is configured to permit rotation of the payload about a yaw axis. The third carrier component is configured to permit rotation of the payload about a roll axis and connects to the payload. Document <CIT> discloses a stabilization mount for a camera that has automatic roll axis compensation, so that the camera gives a steady image when in motion. The mount may be hand-held and has a slidable counterweight, a roll motion sensor, and a motor, responsive to a controller, to provide roll stability. A way to address the above drawback is that the effect of the vehicle roll (hereinafter the "roll effect") in the optical data can be corrected post-capture. For example, the captured optical data can be rotated in a direction opposing the vehicle roll, and then fitted into the aspect ratio of the optical sensor. A drawback of this response is that the rotation and fitting of the optical data results in loss of information for computer vision processing because the rotation and fitting can cause portions of the optical data to be cropped off. This loss of information results in less effective computer vision for driver assistance and/or the like. What is needed are more effective techniques for correcting the roll effect in optical data captured by an optical sensor.

An embodiment sets forth a computer-implemented method comprising receiving sensor data from at least one sensor associated with the vehicle, filtering the sensor data to remove noise from the sensor data, detecting an amount of a roll of the vehicle based on the filtered sensor data, generating a command based on the detected amount of roll, and controlling an orientation of the optical sensor based on the command. The step of filtering the sensor data comprises computing a moving average of a given number of previous data points included in the sensor data, or performing exponential smoothing on the sensor data.

Further embodiments provide, among other things, one or more non-transitory computer-readable media and systems configured to implement the method set forth above.

At least one technical advantage of the disclosed techniques relative to the prior art is that the effect of vehicle roll on forward-facing and/or rear-facing optical data can be compensated for without cropping off information from the optical data. Accordingly, more orientation-appropriate optical data information can be provided to image processing and computer vision systems, resulting in higher recognition accuracy by those image processing and computer vision systems and thereby resulting in more effective driver assistance, autonomous driving, and/or the like. These technical advantages provide one or more technological advancements over prior art approaches.

So that the manner in which the above recited features of the various embodiments can be understood in detail, a more particular description of the inventive concepts, briefly summarized above, can be had by reference to various embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of the inventive concepts and are therefore not to be considered limiting of scope in any way, and that there are other equally effective embodiments.

In the following description, numerous specific details are set forth to provide a more thorough understanding of the various embodiments. However, it will be apparent to one of skilled in the art that the inventive concepts may be practiced without one or more of these specific details.

<FIG> is a conceptual block diagram of an optical sensor compensation system <NUM> configured to implement one or more aspects of the various embodiments. As shown, the optical sensor compensation system <NUM> includes a computing device <NUM>, sensor(s) <NUM>, actuator <NUM>, and optical sensor <NUM>. Computing device <NUM> includes one or more processing unit(s) <NUM>, and memory <NUM>. Memory <NUM> stores a compensation application <NUM> and an actuator controller <NUM>. In some embodiments, sensor(s) <NUM>, actuator <NUM>, optical sensor <NUM>, and computing device <NUM>, are implemented at a vehicle.

Sensor(s) <NUM> can include any type of device that is capable of receiving and/or transmitting sensor data, including for example accelerometer data, angular velocity data, and so forth. More generally, the sensor data can include angular data of the vehicle, including but not limited to angular velocity, angular rotation or displacement relative to one or more axes and/or planes, and/or the like. In some embodiments, sensor(s) <NUM> include one or more sensors that provide other data, such as location data, image data, temperature data, etc. Computing device <NUM> can compute an amount of angular rotation or displacement corresponding to a roll of a vehicle on which sensor(s) <NUM> are mounted based on the acquired sensor data. Non-limiting examples of sensor(s) <NUM> include accelerometers, gyroscopes, magnetometers, computing devices, smartphones, navigation devices, imaging devices, Internet of Things (IoT) devices, radiofrequency identification (RFID) devices, traffic devices, global positioning devices, etc. In various embodiments, sensor(s) <NUM> can communicate with computing device <NUM> via a wired or wireless connection. In some embodiments, sensors <NUM> include an inertial measurement unit (IMU) that includes one or more sensors such as one or more accelerometer(s), one or more gyroscope(s), and/or one or more magnetometer(s).

As noted above, computing device <NUM> includes processing unit(s) <NUM> and memory <NUM>. Computing device <NUM> can be a device that includes one or more processing units <NUM>, such as a system-on-a-chip (SoC). In some embodiments, computing device <NUM> can be a head unit or other component included in a vehicle system. Generally, computing device <NUM> can be configured to coordinate the overall operation of optical sensor compensation system <NUM>. The embodiments disclosed herein contemplate any technically-feasible system configured to implement the functionality of optical sensor compensation system <NUM> via computing device <NUM>.

In various embodiments, computing device <NUM> can be located in various environments including, without limitation, road and/or land vehicle environments (e.g., consumer vehicle, commercial vehicle, bicycle, motorcycle, wheeled drone, etc.), aerospace and/or aeronautical vehicle environments (e.g., airplane, helicopter, spaceship, glider, aerial drone, etc.), nautical and submarine vehicle environments (e.g., boat, yacht, submarine, personal watercraft, nautical or submarine drone, etc.), and so forth.

Processing unit(s) <NUM> can be any technically-feasible form of processing device configured to process data and execute program code. Processing unit(s) <NUM> could include, for example, and without limitation, a system-on-chip (SoC), a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field-programmable gate array (FPGA), and so forth. Processing unit(s) <NUM> includes one or more processing cores. In operation, processing unit <NUM> can be a primary processor of computing device <NUM>, controlling and coordinating operations of other system components.

Memory <NUM> can include a memory module or a collection of memory modules. In various embodiments, processing unit(s) <NUM> can execute compensation application <NUM> and/or actuator controller <NUM> to implement the overall functionality of the computing device <NUM> and, thus, to coordinate the operation of the optical sensor compensation system <NUM> as a whole. For example, and without limitation, sensor data acquired via the sensors <NUM> can be processed by compensation application <NUM> to generate a command. Processing unit(s) <NUM> can execute actuator controller <NUM> to drive actuator <NUM> based on the command generated by compensation application <NUM>. In various embodiments, compensation application <NUM> can control the timing of sending the command to actuator controller <NUM>. For example, compensation application <NUM> can send a command that includes an amount of angular rotation of a vehicle about a longitudinal axis of the vehicle to actuator controller <NUM>, driving actuator <NUM> to orient optical sensor <NUM> by an amount that counters the amount of roll by the vehicle about the longitudinal axis. In some examples, compensation application <NUM> can send the command within <NUM> of receiving the roll data from sensor(s) <NUM>. In some embodiments, compensation application <NUM> and actuator controller <NUM> are stored in a storage medium (not shown) of computing device <NUM> and loaded into memory <NUM> for execution. In some embodiments, compensation application <NUM> and actuator controller <NUM> can be combined into one application. Operation of compensation application <NUM> and actuator controller <NUM> is further described below.

Actuator <NUM> controls the orientation of optical sensor <NUM>. In various embodiments, actuator <NUM> controls the orientation of optical sensor <NUM> along at least one axis (e.g., a longitudinal axis that is parallel to the longitudinal axis of the vehicle). In some embodiments, actuator <NUM> and actuator controller <NUM> can be included in a separate optical sensor orientation system (not shown). In such instances, actuator controller <NUM> can receive a command from computing device <NUM> and drive actuator <NUM> to rotate optical sensor <NUM> according to an angle (e.g., an amount of angular rotation of the vehicle about the longitudinal axis of the vehicle) included in the command. In some embodiments, actuator <NUM> controls one or more stabilizer components (not shown) that modify the orientation of optical sensor <NUM> relative to the road surface. Actuator <NUM> includes various mechanical, electro-mechanical, and/or other components (e.g., gears, actuators, hydraulic systems, pneumatic systems, etc.) that physically change the orientation of optical sensor <NUM>.

Optical sensor(s) <NUM> includes at least one sensor that acquires optical data. In various embodiments, optical sensor <NUM> acquires optical data relating to the environment of a vehicle. In various embodiments, optical sensor <NUM> can include one or more cameras, such as RGB cameras, infrared cameras, thermal cameras, night vision cameras, depth cameras, and/or camera arrays, which include two or more of such cameras. Other optical sensors can include imagers and laser sensors. Further, optical sensor <NUM> can include one or more components of an imaging system, such as one or more components of a RADAR, a LiDAR system, and/or the like. In various embodiments, a vehicle can include one or more optical sensors <NUM>. For example, a vehicle could include a forward-facing optical sensor (for capturing the environment in front of the vehicle) and/or a rear-facing optical sensor (for capturing the environment to the rear of the vehicle).

In various embodiments, optical sensor <NUM> is coupled to actuator <NUM>. For example, optical sensor <NUM> could be mounted on a frame or a shaft that is coupled to and rotatable by actuator <NUM>. In some embodiments, the vehicle could include multiple actuators <NUM> (e.g., an actuator <NUM> for each optical sensor <NUM>).

In various embodiments, optical sensor <NUM> can send optical data to computing device <NUM> and/or another processing system. In such instances, computing device <NUM> and/or the other processing system analyzes the image data. For example, compensation system <NUM> could include an image processing system <NUM> that processes images to identify entities (e.g., objects, people, animals). In such instances, optical sensor <NUM> could send optical data in the form of a captured image to image processing system <NUM>, where image processing system <NUM> analyzes the captured image using any technical feasible technique(s) (e.g., object recognition, computer optical vision, machine learning, etc.) in order to identify entities in the image. Image processing system <NUM> could send image analysis results (e.g., the objects recognized in the image) to a vehicle control assistance system <NUM>. Image processing system <NUM> can be implemented via hardware, software, and/or a combination of hardware and software.

In some embodiments, compensation system <NUM> further includes a vehicle control assistance system <NUM>. Vehicle control assistance system <NUM> can include one or more applications or sub-systems that provides assistance or take-over (e.g., information to a human driver, automated or autonomous vehicle operation) regarding operation of the vehicle, such as a driver assistance system, an autonomous driving system, an automated cruise control system, a collision avoidance system, and/or the like. Vehicle control assistance system <NUM> Vehicle control assistance system <NUM> can use the image analysis results to provide assistance or take-over based on the results (e.g., an alert to a human driver, autonomously brake and/or steer the vehicle). Vehicle control assistance system <NUM> can be implemented via hardware, software, and/or a combination of hardware and software. Image processing system <NUM> and/or vehicle control assistance system <NUM> can be an application or a sub-system within computing device <NUM> or implemented elsewhere in the vehicle (e.g., another application or sub-system in the vehicle head unit or in-vehicle computing system). In some embodiments, image processing system <NUM> and vehicle control assistance system <NUM> can be combined.

A vehicle can include a driver assistance system that helps a user operate the vehicle safely and assists in certain tasks associated with the vehicle, enabling the user to enjoy the ride more. Additionally or alternatively, a vehicle can include an autonomous driving system that operates the vehicle with little or no driver intervention. Computer optical vision systems have been key components in driver assistance systems and other vehicle systems, such as autonomous systems for four-wheel cars and trucks. When included in a driver assistance system, the computer optical vision system helps the user by providing feedback. Such feedback can include information about how quickly to enter a corner, or an alert to notify a driver who is not paying close attention to the road. In autonomous systems, the computer optical vision system enables the autonomous driving system to safely navigate an environment based on data acquired about the environment.

A computer optical vision system includes one or more optical sensors (e.g., still cameras, video cameras, LiDAR systems, etc.) that captures images of the surrounding environment. The computer optical vision system includes an image processing system that receives the one or more images captured by the optical sensors as an input and processes the image in order to operate the vehicle. For example, an autonomous vehicle could employ a computer optical vision system to accurately detect entities (e.g., vehicles, street buildings, pedestrians, road signs, etc.) in order to assist the autonomous vehicle to drive as safely as a human. Similarly, a driver assistance system could detect entities and notify a human driver of potential hazards.

<FIG> are diagrams illustrating principal axes for a four-wheel vehicle and a motorcycle, according to various embodiments. As shown in <FIG>, a four-wheel vehicle <NUM> has a lateral axis, a longitudinal axis, and a vertical axis. Four-wheel vehicle <NUM> rarely rolls or rarely rolls appreciably around the longitudinal axis during operation. As used herein, a roll of a vehicle is a rotation of the vehicle around the longitudinal axis, thereby causing an amount of angular rotation relative and orthogonal to a vertical axis of the vehicle, where that vertical axis is perpendicular to a horizontal plane and/or a plane of the road surface on which the vehicle is travelling. Instead, vehicle <NUM> typically rotates (i.e., is steered) around the vertical axis (e.g., the z-axis or heave axis). Vehicle <NUM> can also experience pitch around the lateral axis relative to a horizontal plane (e.g., when travelling uphill or downhill) and/or relative to the plane of the road surface. The vehicle can provide yaw control by controlling the motion around the vertical axis. Thus, optical sensors mounted to the front and/or rear of a four-wheel vehicle (e.g., where the field of view of the optical sensor is directed toward the front or rear of the vehicle) with a view-up orientation that is upright and parallel to the vertical axis of the vehicle have a view-up orientation that remains substantially perpendicular to the horizontal plane and/or the road surface. Image processing system <NUM> generally accurately detects entities in such images.

<FIG> illustrates an example image of a road environment captured without a roll effect, according to various embodiments. As shown, image <NUM> is an image captured by a forward-facing optical sensor mounted on a vehicle. Image <NUM> is captured while the vehicle has no appreciable roll , and accordingly the view-up orientation of the optical sensor at the moment of capture, and correspondingly the view-up orientation of image <NUM>, is substantially perpendicular to the road surface. Accordingly, image <NUM> is substantially free of a roll effect.

Image <NUM> has a predefined aspect ratio <NUM>. Image <NUM> includes a road <NUM> in front of the vehicle. A vehicle <NUM> is on road <NUM> in front of the vehicle. Another vehicle <NUM> is on the opposite road, across a median <NUM>. A tree <NUM> is located on the median <NUM>. A traffic light <NUM> is located on the side of road <NUM>. Image processing system <NUM> processing image can analyze image <NUM> and accurately recognize and detect vehicles <NUM> and <NUM>, tree <NUM>, and traffic light <NUM>, as indicated by recognition bounding boxes <NUM>.

Computer optical vision systems can also be implemented in other vehicles that provide controls along one or two other axes. For example, returning to <FIG>, a motorcycle <NUM> also has a lateral axis, a longitudinal axis, and a vertical axis. Vehicles such as motorcycle <NUM>, submarines, aerial vehicles (including quadcopters), boats, and so forth, can roll around the longitudinal axis. Vehicle <NUM> can provide roll control to manage the amount of roll. Thus, the view-up orientations of optical sensors mounted to the front and/or rear of vehicle <NUM>, mounted with a view-up orientation that is upright and parallel to the vertical axis of vehicle <NUM>, are not perpendicular to the road surface while vehicle <NUM> is rolled about the longitudinal axis of vehicle <NUM>.

Because a typical image processing system is trained using images where the view-up orientation of the optical sensor is substantially perpendicular to the road surface and/or the horizontal plane, the image processing system detects entities less accurately in images where the view-up orientation of the optical sensor is not perpendicular to the road surface and/or the horizontal plane. While such images can be corrected post-capture (e.g., by rotating the images and then fitting the images into the predefined aspect ratio required by the image processing system), such correction has the drawback of causing portions of the original image to be cropped off, resulting in less information for the image processing system.

<FIG> illustrates an example image of the road environment captured with a roll effect, according to various embodiments. As shown, image <NUM> is an image of the same road environment as in image <NUM>, captured by the same forward-facing optical sensor as with image <NUM>. Image <NUM> is captured while the vehicle is rolled counter-clockwise about the longitudinal axis the vehicle relative to the plane of the surface of road <NUM>, and accordingly the view-up orientation of the optical sensor at the moment of capture, and correspondingly the view-up orientation of image <NUM>, is not perpendicular to the road surface but is angled. Accordingly, image <NUM> has a roll effect.

Image <NUM> has a predefined aspect ratio <NUM> that is the same as aspect ratio <NUM>. Shown in image <NUM> is road <NUM> in front of the vehicle. Image <NUM> includes vehicle <NUM> on road <NUM> in front of the vehicle. Image <NUM> also includes a portion of vehicle <NUM> on the opposite road across a median <NUM>, and a portion of tree <NUM> on median <NUM>. A traffic light <NUM> is located on the side of road <NUM>.

<FIG> illustrates the example image of <FIG> with roll effect correction according to conventional techniques. <FIG> illustrates an image <NUM> that can be generated by rotating image <NUM> opposite the direction of the roll of the vehicle and then fitting the rotated image into aspect ratio <NUM>, which is the same aspect ratio as aspect ratio <NUM>/<NUM>. Thus, the view-up orientation of image <NUM> is substantially perpendicular to the surface of road <NUM>. Because the rotated image is fitted into aspect ratio <NUM>, portions <NUM> of the rotated image <NUM> that are outside of aspect ratio <NUM> are cropped off and not included in image <NUM>. Instead, image <NUM> includes portions <NUM> that are blank (e.g., fully white or black) and correspond to portions of the aspect ratio <NUM> that are not covered by any part of the rotated image <NUM> fitted into aspect ratio <NUM> and are instead filled with blank content (e.g., a single color). Accordingly, image <NUM> contains less information than image <NUM> or <NUM>. An image processing system can process image <NUM> and, because of the reduced amount of information in the image, detect entities less accurately and/or detect fewer entities in the image. For example, as shown in <FIG>, image processing system <NUM> would detect just vehicle <NUM> and traffic light <NUM> in image <NUM>, indicated by bounding boxes <NUM>. Meanwhile, significant portions of vehicle <NUM> and tree <NUM> are not captured in image <NUM> and are accordingly not detected by image processing system <NUM>.

To address the above drawbacks, optical sensor compensation system <NUM> performs roll effect correction prior to capture of an image. Compensation application <NUM> acquires sensor data from sensors <NUM>, determines an amount of roll of the vehicle (e.g., an amount of angular rotation around the longitudinal axis of the vehicle), and generates an actuator command based on the amount of roll. Compensation application <NUM> sends the actuator command to actuator controller <NUM>, which drives actuator <NUM> based on the actuator command to modify the orientation of optical sensor <NUM> to counteract the amount of roll. Optical sensor <NUM>, with the modified orientation, can capture one or more images, which can be processed by image processing system <NUM>.

In operation, compensation application <NUM> acquires sensor data from sensors <NUM>. Compensation application <NUM> receives sensor data that can be used to detect an amount of roll of the vehicle about the longitudinal axis of the vehicle from sensors <NUM> (e.g., from an IMU and/or an accelerometer). The sensor data can include a roll angle parameter that directly measures an amount of roll (e.g., the amount of angular rotation relative to the axis perpendicular to the road surface) and a direction of roll (e.g., rolling clockwise or counterclockwise about the longitudinal axis of the vehicle relative to the vertical axis). Additionally or alternatively, the sensor data can include one or more parameters (e.g., angle relative to a horizontal plane, angular velocity, etc.) that can indirectly indicate rolling and can be used to calculate the amount and direction of roll.

Compensation application <NUM> can determine an amount and direction of roll based on the sensor data and accordingly determine or detect whether the vehicle is rolling. For example, if the sensor data includes a roll angle parameter, compensation application <NUM> can detect whether the vehicle is rolling from the roll angle parameter. Additionally or alternatively, compensation application <NUM> can calculate the amount and direction of roll from the sensor data and detect whether the vehicle is rolling from the calculated amount and direction of roll. In some embodiments, compensation application <NUM> detects rolling that exceeds a threshold and disregards rolling that does not exceed the threshold. For example, compensation application <NUM> could disregard amounts of roll that are smaller than a threshold (e.g., angular rotation of less than <NUM> degrees relative to the vertical axis).

Compensation application <NUM> filters the sensor data as part of the determination of the roll angle and direction. In some other embodiments, sensors <NUM> (e.g., accelerometer, IMU) can filter the sensor data before sending the sensor data (e.g., the roll amount and direction) to compensation application <NUM>. The filtering can be performed with a low-pass filter, for example. In these embodiments, filtering the sensor data can remove noise amongst the sensor data and provide a more-accurate determination of the amount of roll. For example, sensors <NUM> could generate noise within the sensor data due to vibrations and jerks felt by the vehicle (e.g., driving over an uneven road surface).

To remove noise, according to embodiments of the disclosure, sensors <NUM> and/or compensation application <NUM> compute a simple moving average for incoming sensor data. For example, when sensors <NUM> filter incoming sensor data, with each iteration in a code loop, sensors <NUM> could drop the oldest value from the previous n datapoints and replaces that value with the latest reading. Sensors <NUM> can use Equation <NUM> to compute an average for the amount of roll (e.g., angular rotation of the roll).

Where dsm is the simple moving average at current data point dm, and n is the number of previous data points taken into consideration. In one example, sensors <NUM> or compensation application <NUM> can compute the angular rotation of the roll as an average of the previous twenty data points acquired by the accelerometer. In some embodiments, sensors <NUM> and/or compensation application <NUM> perform exponential smoothing on the sensor data in lieu of calculating a simple moving average. More generally, sensors <NUM> and/or compensation application <NUM> can aggregate, using any suitable algorithm or technique, any number of roll angle values in the sensor data before optical sensor <NUM> captures an image to determine an amount and direction of roll, so that the orientation of optical sensor <NUM> can be modified before the image is captured.

Using the new reading, sensors <NUM> can compute a new aggregate value. This new aggregate value is sent to compensation application <NUM>, which uses the new aggregate value to generate an actuator command for actuator controller <NUM>. Using aggregated values to drive the motion controller can ensure smooth operation of the motor.

In some embodiments, compensation application <NUM> acquires sensor data at a more frequent rate than the frame rate of optical sensor <NUM>. For example, the frame rate of optical sensor <NUM> could be <NUM> frames per second, and sensors <NUM> could provide sensor data to compensation application <NUM> at a rate of <NUM> data points per second. When determining an amount and direction of roll before capture of a next frame by optical sensor <NUM>, compensation application <NUM> can use the data points of sensor data acquired between that next frame and the immediately preceding frame of optical sensor <NUM>. Those data points can be optionally processed (e.g., aggregated, average, moving average, exponentially smoothed) to determine the amount and direction of roll.

After determining the amount and direction of roll, compensation application <NUM> can generate an actuator command. The actuator command can include an angle by which to rotate optical sensor <NUM> and a direction of rotation. The angle and direction included in the actuator command are generated to counter the amount and direction of roll of the vehicle. For example, if the vehicle is rolling counterclockwise <NUM> degrees, the actuator command would include an angle of <NUM> degrees clockwise by which to rotate a forward-facing optical sensor <NUM>. Compensation application <NUM> sends the actuator command to actuator controller <NUM>. In some embodiments, if compensation application <NUM> detects that the vehicle is not rolling (e.g., the amount of roll is below a threshold or is otherwise zero or minimal), compensation application <NUM> can omit generating an actuator command.

Actuator controller <NUM> receives the actuator command and drives actuator <NUM> based on the actuator command. Actuator controller <NUM> activates actuator <NUM> to rotate optical sensor <NUM> by the angle and direction included in the actuator command, thereby re-orienting optical sensor <NUM> so that the view-up orientation of optical sensor <NUM> is substantially perpendicular to the road surface and/or the horizontal plane.

In some embodiments, compensation application <NUM> and actuator controller <NUM> can perform the roll effect correction process for multiple optical sensors. For example, if the vehicle includes a forward-facing optical sensor and a rear-facing optical sensor, compensation application <NUM> could determine the amount and direction of roll of the vehicle, and then determine respective actuator commands for the forward-facing optical sensor and the rear-facing optical sensor. Actuator controller <NUM> could drive respective actuators <NUM> based on the respective actuator commands. Additionally or alternatively, compensation application <NUM> can generate a command, and the actuator controller <NUM> drives the respective actuators <NUM> based on the command, with actuator controller <NUM> being responsible for interpreting the command into the proper amounts and directions of rotation based on whether the particular actuator is rotating a forward-facing optical sensor or a rear-facing optical sensor.

<FIG> illustrates a conceptual diagram of roll effect correction by modifying an orientation of an optical sensor, according to various embodiments. <FIG> illustrates a front view of a motorcycle <NUM> that can roll about its longitudinal axis. Vehicle <NUM> has a forward-facing optical sensor <NUM> (e.g., a camera <NUM>) that faces out toward the front environment of motorcycle <NUM> with a field of view aligned parallel to the longitudinal axis of motorcycle <NUM>. When motorcycle <NUM> rolls, motorcycle <NUM> rolls by an angular rotation amount Θx <NUM> and in a direction (e.g., clockwise from motorcycle <NUM> facing forward as shown) relative to a vertical axis <NUM> that is perpendicular to road surface <NUM>. Compensation application <NUM> can determine the angular rotation amount Θx <NUM> and the direction of roll, and generate a corresponding actuator command. Actuator controller <NUM> can drive an actuator <NUM> on-board motorcycle <NUM> to rotate <NUM> camera <NUM> by angular rotation amount Θx and in a direction (e.g., counterclockwise from camera <NUM> facing forward as shown) that counters the direction of the roll of motorcycle <NUM>, thereby maintaining the view-up orientation of camera <NUM> to be perpendicular to road surface <NUM>.

It should be appreciated that the angular rotation amount of the roll can be expressed as an angle relative to the vertical axis or as an angle relative to a horizontal plane or a plane of the road surface. In various embodiments, sensors <NUM> and/or compensation application <NUM> can determine an angle relative to the vertical axis from an angle relative to a horizontal plane or a plane of the road surface, or vice versa.

<FIG> illustrates a flow diagram of method steps to compensate for the roll effect for an optical sensor, according to various embodiments. Although the method steps are described with respect to the systems of <FIG>, persons skilled in the art will understand that any system configured to perform the method steps, in any order, falls within the scope of the various embodiments.

As shown, a method <NUM> begins at step <NUM>, where compensation application <NUM> acquires sensor data from one or more sensors <NUM> (e.g., IMU, accelerometer) on a vehicle.

At step <NUM>, compensation application <NUM> determines an amount and direction of roll of the vehicle. Compensation application <NUM> can read the data points in the sensor data and determine an amount of roll and a direction of roll directly from the sensor data (e.g., from a roll angle parameter in the sensor data). Additionally or alternatively, compensation application <NUM> can calculate an amount and direction of roll from the sensor data. Further additionally or alternatively, compensation application <NUM> can determine an amount and direction of roll from multiple data points in the sensor data. In some embodiments, sensors <NUM> and/or compensation application <NUM> can aggregate and/or filter data points in the sensor data before determining the amount and direction of roll. In some embodiments, compensation application <NUM> can determine if the amount of roll is less than a threshold amount (e.g., is zero, is less than a threshold amount of roll). If compensation application <NUM> determines that the amount of roll is less than the threshold amount, then compensation application <NUM> can omit steps <NUM> and <NUM> below.

At step <NUM>, compensation application <NUM> generates an actuator command based on the amount and direction of roll. The actuator command includes an angle and a direction that is intended to counter the amount and direction of roll of the vehicle determined in step <NUM>. For example, the actuator command can include an amount of rotation equal to the amount of roll of the vehicle, but in a direction opposite to the direction of roll of the vehicle. If compensation application <NUM> determines that the amount of roll is less than the threshold amount, then compensation application <NUM> can, instead of omitting steps <NUM> and <NUM> as described above, generate a null or no-op command.

At step <NUM>, compensation application <NUM> causes an actuator to modify an orientation of an optical sensor on the vehicle based on the actuator command. Compensation application <NUM> sends the actuator command generated in step <NUM> to an actuator controller <NUM>. Actuator controller <NUM> drives an actuator <NUM> based on the actuator command to modify an orientation of an optical sensor <NUM> (e.g., rotating optical sensor <NUM> to maintain a view-up orientation perpendicular to the road surface). If the command is a null or no-op command, actuator controller <NUM> can receive the null or no-op command and take no action in response.

At step <NUM>, computing device <NUM> captures an image using the optical sensor. Computing device <NUM> and/or image processing system <NUM> can capture an image using optical sensor <NUM> whose orientation is modified in step <NUM>. At step <NUM>, image processing system <NUM> processes the captured image to recognize one or more entities in the image. Image processing system <NUM> in computing device <NUM> or elsewhere in the vehicle can process the image captured in step <NUM> to recognize one or more objects, etc. in the image. Image processing system <NUM> can send the processing results to a vehicle control assistance system <NUM>.

In sum, the disclosed techniques compensate for the amount and direction of roll in a structure (e.g., a vehicle) when capturing optical data via a forward-facing or rear-facing optical sensor mounted on the vehicle. The techniques include acquiring sensor data from sensors on the vehicle. A compensation application processes the sensor data to determine an amount and direction of roll of the vehicle and to generate an actuator command based on the determined amount and direction of roll. An actuator controller drives an actuator based on the actuator command to modify the orientation of a forward-facing or rear-facing optical sensor mounted on the vehicle and coupled to the actuator, thereby re-orienting the view-up orientation of the optical sensor to be substantially perpendicular to the road surface while the vehicle is undergoing roll. The optical sensor whose orientation is modified by the disclosed techniques can then capture optical data, and those optical data, which exhibit reduced or eliminated roll effect due to the modified orientation of the optical sensor, can be provided to an image processor for processing.

Aspects of the present embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "module," a "system," or a "computer. " In addition, any hardware and/or software technique, process, function, component, engine, module, or system described in the present disclosure may be implemented as a circuit or set of circuits. Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

Claim 1:
A computer-implemented method for controlling an optical sensor (<NUM>) mounted on a vehicle, comprising:
receiving sensor data from at least one sensor (<NUM>) associated with the vehicle;
filtering the sensor data to remove noise from the sensor data;
detecting an amount of roll of the vehicle based on the filtered sensor data;
generating a command based on the detected amount of roll; and
controlling an orientation of the optical sensor (<NUM>) based on the command;
characterised in that the step of filtering the sensor data comprises: computing a moving average of a given number of previous data points included in the sensor data; or
performing exponential smoothing on the sensor data.