Patent ID: 12260593

DETAILED DESCRIPTION

Various examples of the present technology are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without parting from the spirit and scope of the present technology. In some instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more aspects. Further, it is to be understood that functionality that is described as being carried out by certain system components may be performed by more or fewer components than shown.

The disclosed technologies address a need in the art for improvements to vehicle sensor calibration technologies. Use of a dynamic calibration scene with an automated turntable or carousel system improves the functioning of sensor calibration by improving runtime-efficiency, space-efficiency, comprehensiveness of calibration, and consistency of vehicle sensor calibration. The described vehicle sensor calibration technologies ultimately transform vehicle sensors from an uncalibrated state to a calibrated state. The described vehicle sensor calibration technologies are implemented using a vehicle, the vehicle's sensors, a turntable and other potential components of a dynamic scene, one or more computing devices associated with the other components, each of which is integral at least one embodiment of the vehicle sensor calibration technologies.

Sensors coupled to a vehicle are calibrated, optionally using a dynamic scene with sensor targets around a motorized turntable that rotates the vehicle to different orientations. One vehicle sensor captures a representation of one feature of a sensor target, while another vehicle sensor captures a representation of a different feature of the sensor target, the two features of the sensor target having known relative positioning on the target. The vehicle generates a transformation that maps the captured representations of the two features to positions around the vehicle based on the known relative positioning of the two features on the target.

FIG.1illustrates an autonomous vehicle and remote computing system architecture.

The autonomous vehicle102can navigate about roadways without a human driver based upon sensor signals output by sensor systems180of the autonomous vehicle102. The autonomous vehicle102includes a plurality of sensor systems180(a first sensor system104through an Nth sensor system106). The sensor systems180are of different types and are arranged about the autonomous vehicle102. For example, the first sensor system104may be a camera sensor system and the Nth sensor system106may be a Light Detection and Ranging (LIDAR) sensor system. Other exemplary sensor systems include radio detection and ranging (RADAR) sensor systems, Electromagnetic Detection and Ranging (EmDAR) sensor systems, Sound Navigation and Ranging (SONAR) sensor systems, Sound Detection and Ranging (SODAR) sensor systems, Global Navigation Satellite System (GNSS) receiver systems such as Global Positioning System (GPS) receiver systems, accelerometers, gyroscopes, inertial measurement units (IMU), infrared sensor systems, laser rangefinder systems, ultrasonic sensor systems, infrasonic sensor systems, microphones, or a combination thereof. While four sensors180are illustrated coupled to the autonomous vehicle102, it should be understood that more or fewer sensors may be coupled to the autonomous vehicle102.

The autonomous vehicle102further includes several mechanical systems that are used to effectuate appropriate motion of the autonomous vehicle102. For instance, the mechanical systems can include but are not limited to, a vehicle propulsion system130, a braking system132, and a steering system134. The vehicle propulsion system130may include an electric motor, an internal combustion engine, or both. The braking system132can include an engine brake, brake pads, actuators, and/or any other suitable componentry that is configured to assist in decelerating the autonomous vehicle102. In some cases, the braking system132may charge a battery of the vehicle through regenerative braking. The steering system134includes suitable componentry that is configured to control the direction of movement of the autonomous vehicle102during navigation.

The autonomous vehicle102further includes a safety system136that can include various lights and signal indicators, parking brake, airbags, etc. The autonomous vehicle102further includes a cabin system138that can include cabin temperature control systems, in-cabin entertainment systems, etc.

The autonomous vehicle102additionally comprises an internal computing system110that is in communication with the sensor systems180and the systems130,132,134,136, and138. The internal computing system includes at least one processor and at least one memory having computer-executable instructions that are executed by the processor. The computer-executable instructions can make up one or more services responsible for controlling the autonomous vehicle102, communicating with remote computing system150, receiving inputs from passengers or human co-pilots, logging metrics regarding data collected by sensor systems180and human co-pilots, etc.

The internal computing system110can include a control service112that is configured to control operation of the vehicle propulsion system130, the braking system208, the steering system134, the safety system136, and the cabin system138. The control service112receives sensor signals from the sensor systems180as well communicates with other services of the internal computing system110to effectuate operation of the autonomous vehicle102. In some embodiments, control service112may carry out operations in concert one or more other systems of autonomous vehicle102.

The internal computing system110can also include a constraint service114to facilitate safe propulsion of the autonomous vehicle102. The constraint service116includes instructions for activating a constraint based on a rule-based restriction upon operation of the autonomous vehicle102. For example, the constraint may be a restriction upon navigation that is activated in accordance with protocols configured to avoid occupying the same space as other objects, abide by traffic laws, circumvent avoidance areas, etc. In some embodiments, the constraint service can be part of the control service112.

The internal computing system110can also include a communication service116. The communication service can include both software and hardware elements for transmitting and receiving signals from/to the remote computing system150. The communication service116is configured to transmit information wirelessly over a network, for example, through an antenna array that provides personal cellular (long-term evolution (LTE), 3G, 4G, 5G, etc.) communication.

In some embodiments, one or more services of the internal computing system110are configured to send and receive communications to remote computing system150for such reasons as reporting data for training and evaluating machine learning algorithms, requesting assistance from remoting computing system or a human operator via remote computing system150, software service updates, ridesharing pickup and drop off instructions etc.

The internal computing system110can also include a latency service118. The latency service118can utilize timestamps on communications to and from the remote computing system150to determine if a communication has been received from the remote computing system150in time to be useful. For example, when a service of the internal computing system110requests feedback from remote computing system150on a time-sensitive process, the latency service118can determine if a response was timely received from remote computing system150as information can quickly become too stale to be actionable. When the latency service118determines that a response has not been received within a threshold, the latency service118can enable other systems of autonomous vehicle102or a passenger to make necessary decisions or to provide the needed feedback.

The internal computing system110can also include a user interface service120that can communicate with cabin system138in order to provide information or receive information to a human co-pilot or human passenger. In some embodiments, a human co-pilot or human passenger may be required to evaluate and override a constraint from constraint service114, or the human co-pilot or human passenger may wish to provide an instruction to the autonomous vehicle102regarding destinations, requested routes, or other requested operations.

The internal computing system110can, in some cases, include at least one computing system1500as illustrated in or discussed with respect toFIG.15, or may include at least a subset of the components illustrated inFIG.15or discussed with respect to computing system1500.

As described above, the remote computing system150is configured to send/receive a signal from the autonomous vehicle140regarding reporting data for training and evaluating machine learning algorithms, requesting assistance from remote computing system150or a human operator via the remote computing system150, software service updates, rideshare pickup and drop off instructions, etc.

The remote computing system150includes an analysis service152that is configured to receive data from autonomous vehicle102and analyze the data to train or evaluate machine learning algorithms for operating the autonomous vehicle102. The analysis service152can also perform analysis pertaining to data associated with one or more errors or constraints reported by autonomous vehicle102.

The remote computing system150can also include a user interface service154configured to present metrics, video, pictures, sounds reported from the autonomous vehicle102to an operator of remote computing system150. User interface service154can further receive input instructions from an operator that can be sent to the autonomous vehicle102.

The remote computing system150can also include an instruction service156for sending instructions regarding the operation of the autonomous vehicle102. For example, in response to an output of the analysis service152or user interface service154, instructions service156can prepare instructions to one or more services of the autonomous vehicle102or a co-pilot or passenger of the autonomous vehicle102.

The remote computing system150can also include a rideshare service158configured to interact with ridesharing applications170operating on (potential) passenger computing devices. The rideshare service158can receive requests to be picked up or dropped off from passenger ridesharing app170and can dispatch autonomous vehicle102for the trip. The rideshare service158can also act as an intermediary between the ridesharing app170and the autonomous vehicle wherein a passenger might provide instructions to the autonomous vehicle to102go around an obstacle, change routes, honk the horn, etc.

The rideshare service158as depicted inFIG.1illustrates a vehicle102as a triangle en route from a start point of a trip to an end point of a trip, both of which are illustrated as circular endpoints of a thick line representing a route traveled by the vehicle. The route may be the path of the vehicle from picking up the passenger to dropping off the passenger (or another passenger in the vehicle), or it may be the path of the vehicle from its current location to picking up another passenger.

The remote computing system150can, in some cases, include at least one computing system1500as illustrated in or discussed with respect toFIG.15, or may include at least a subset of the components illustrated inFIG.15or discussed with respect to computing system1500.

FIG.2Aillustrates a camera calibration target with a checkerboard pattern on a planar substrate.

The sensor calibration target220A illustrated inFIG.2Ais a planar board made from a substrate205, with a pattern210A printed, stamped, engraved, imprinted, or otherwise marked thereon. The pattern210A ofFIG.2Ais a checkerboard pattern. The substrate205may be paper, cardboard, plastic, metal, foam, or some combination thereof. The substrate205may in some cases include a translucent or transparent surface upon which the pattern210A is printed, and which a light source may provide illumination through. The substrate205may in some cases include a retroreflective surface upon which the pattern210A is printed. The retroreflective property of the surface may be inherent to the material of the substrate205or may be a separate layer applied to the surface of the substrate, for example by adhering a retroreflective material to the substrate205or by painting (e.g., via a brush, roller, or aerosol spray) the substrate205with a retroreflective paint. A reflective or retroreflective property may in some cases improve detection using radar, lidar, or other EmDAR sensors. The material and shape of the substrate205may also be selected such that the material and/or shape produces a high amount of acoustic resonance or acoustic response to improve detection using SONAR or SODAR sensors. In some cases, the substrate205, and therefore the target200A, may be concave, convex, otherwise curved, or some combination thereof. The substrate205may in some cases include devices, such as speakers, heat sources, or light sources, that allow improved detection by microphones, infrared sensors, or cameras, respectively.

The sensor calibration target220A illustrated inFIG.2Ais useful for calibration of a camera of the vehicle, or other sensor that captures visual data. In particular, a camera with a pattern/image/feature recognition system running on computer system110can identify the checkerboard pattern210A ofFIG.2A, and can identify points representing vertices between the dark (black) and light (white) checkers. By drawing lines connecting these points, the camera and computer system110can generate a grid. If the camera has a wide-angle lens, such as a fisheye lens or a barrel lens, the resulting grid will be warped so that some checkers will appear curved rather than straight, and so that checkers near the edges of the camera's point of view will appear more squashed, while checkers near the center of the camera's point of view will appear larger and more even. A rectilinear lens provides a similar, is opposite, effect. Based on prior knowledge of what the checkerboard pattern and resulting grid should look like, and its original dimensions, compared against what its representation looks like as captured by the camera, the camera and computing system110may identify the effect of the lens and counteract it. The camera and computing system110may also identify other parameters of the camera this way, such as position parameters (x, y, z, roll, pitch, yaw), any lens color to be filtered out, any crack or defect in the lens to be filtered out, or a combination thereof.

The sensor calibration target220A illustrated inFIG.2Ais useful for detection by, and calibration of, a range sensor of the vehicle, such as a LIDAR, SONAR, SODAR, or radar sensor of the vehicle, at least in that the shape of the planar substrate205can be detected by the range sensor. For example, flat planar vision targets such as the target220A can be detected by lidar by relying on planar geometry estimates and using the returned intensity. WhileFIG.2Aillustrates a square or rectangular substrate205, the substrate205may be circular, semicircular, ellipsoidal, triangular, quadrilateral (trapezoid, parallelogram), pentagonal, hexagonal, heptagonal, octagonal, nonagonal, decagonal, otherwise polygonal, or some combination thereof.

FIG.2Billustrates a camera calibration target with a ArUco pattern on a planar substrate.

The sensor calibration target220B illustrated inFIG.2B, like the sensor calibration target220A illustrated inFIG.2A, includes a planar board made from a substrate205, with a pattern210B printed, stamped, engraved, imprinted, or otherwise marked thereon. The pattern210B illustrated inFIG.2Bis an ArUco marker pattern, which includes black border and an inner binary matrix/grid (e.g., each square is dark/black or light/white) which determines its identifier.

By detecting the ArUco pattern, the camera and computing system110of the vehicle also identify a grid, similarly to the checkerboard, though potentially with fewer points, as some areas of the ArUco pattern may include contiguous dark/black squares or contiguous light/white squares. By identifying the grid from the representation of the ArUco target captured by the camera (e.g. with lens distortion such as parabolic distortion), and comparing it to a known reference image of the ArUco pattern (e.g., without any distortion), any distortions or other differences may be identified, and appropriate corrections may be generated to counteract these distortions or other differences.

The substrate205ofFIG.2Bmay include or be coated with any previously-discussed substrate material and may be warped or shaped in any manner or include any devices discussed with respect to the substrate205ofFIG.2A, and therefore may be detected by, and be useful to calibrate a range sensor of the vehicle, such as a LIDAR, SONAR, SODAR, or radar sensor of the vehicle, and may be detected by a microphone or infrared sensor of the vehicle as well.

FIG.2Cillustrates a camera calibration target with a crosshair pattern on a planar substrate.

The sensor calibration target220C illustrated inFIG.2B, like the sensor calibration target220A illustrated inFIG.2A, includes a planar board made from a substrate205, with a pattern210C printed, stamped, engraved, imprinted, or otherwise marked thereon. The pattern210C illustrated inFIG.2Cis an crosshair marker pattern, which includes four dark/black lines and two dark/black circles centered on a light/white background, and with a gap in the dark/black lines near but not at the center, effectively leaving a “+” symbol in the very center.

The camera and computing system110can identify the target200C by identifying the circles, the lines, and the intersections of the same. In doing so, the crosshair pattern is identified from the representation of the target220C captured by the camera (e.g. with lens distortion), and can be compared it to a known reference image of the crosshair pattern target200C (e.g., without any distortion). As with the checkerboard and ArUco targets, any distortions or other differences may be identified, and appropriate corrections may be generated to counteract these distortions or other differences.

The substrate205ofFIG.2Cmay include or be coated with any previously-discussed substrate material and may be warped or shaped in any manner or include any devices discussed with respect to the substrate205ofFIG.2A, and therefore may be detected by, and be useful to calibrate a range sensor of the vehicle, such as a LIDAR, SONAR, SODAR, or radar sensor of the vehicle, and may be detected by a microphone or infrared sensor of the vehicle as well.

While the only patterns210A-C discussed with respect to camera sensor targets are checkerboard patterns210A, ArUco patterns210B, and crosshair patterns210C, other patterns that are not depicted can additionally or alternatively be used. For example, bar codes or quick response (QR) codes may be used as patterns210that can be recognized using the camera and computing device110.

FIG.2Dillustrates a range sensor calibration target with a trihedral shape.

The sensor calibration target220ofFIG.2Dis made to be detected by, and use for calibration of, a range sensor, such as a radar sensor (or LIDAR, SONAR, or SODAR) of the vehicle. In particular, the sensor calibration target220ofFIG.2Dis trihedral in shape, and may be a concave or convex trihedral corner, essentially a triangular corner of a cube. Alternately, it may be a different shape, such as a corner of a different polyhedron (at least portions of all faces of the polyhedron that touch a particular vertex). Such a shape, especially when concave and where perpendicular faces are included, produces a particularly strong radar echo and thus a particularly strong radar cross section (RCS) because incoming radio waves are backscattered by multiple reflection. The RCS of the trihedral corner target is given by: σ=(4·π·a4)/(3·λ2), where a is the length of the side edges of the three triangles, and λ is a wavelength of radar transmitter.

The substrate205of the sensor calibration target220ofFIG.2Dmay include or be coated with any previously-discussed substrate material and may be warped or shaped in any manner, or include any devices, discussed with respect to the substrate205ofFIG.2A. In one embodiment, the substrate205of the sensor calibration target220ofFIG.2Dmay be metal, may be electrically conductive, may be reflective or retroreflective, or some combination thereof.

FIG.2Eillustrates a combined range sensor and camera calibration target with apertures from a planar substrate that are surrounded by visually recognizable markings.

The combined range sensor and camera calibration target250ofFIG.2Eincludes multiple apertures225in a substrate205, and includes visual markings or patterns230at least partially surrounding each aperture. In particular, the target250includes four symmetrical circular or ellipsoid apertures225from a light/white substrate205with symmetrical dark/black circular or ellipsoid rings230around the apertures, with three of the apertures/ring combinations being a first size (e.g., apertures being 30 cm in diameter and the corresponding rings slightly larger) and a fourth aperture/ring combination240being a second size (e.g., the aperture being 26 cm in diameter and the corresponding ring slightly larger). The rings around the three larger apertures225are likewise larger than the ring around the smaller aperture240. In some cases, one may be larger than the other three, or two may be larger or smaller than the other two, or some combination thereof. The apertures225/240may alternately be referred to as cutouts, holes, voids, orifices, vents, openings, gaps, perforations, interstices, discontinuities or some combination thereof. In some cases, different types of surface discontinuities may be used instead of or in addition to the apertures225/240, such as raised surfaces or bumps that can also be detected by depth/range sensors such as radar or lidar.

The combined range sensor and camera calibration target250ofFIG.2Ealso includes additional markings or patterns at certain edges of the substrate, identified as target identification (ID) markers235. The particular combined range sensor and camera calibration target250ofFIG.2Eincludes target identification (ID) markers235on the two sides of the substrate opposite the smaller aperture/ring combination240, but other combined range sensor and camera calibration targets250may have one, two, three, or four target identification (ID) markers235along any of the four sides of the square substrate, or may have target identification (ID) markers235in an amount up to the number of sides of the polygonal substrate205where the substrate205is shaped like a non-quadrilateral polygon. That is, if the substrate205is an octagon, each combined combined range sensor and camera calibration target250may have anywhere from zero to eight target identification (ID) markers235. Different patterns of target identification (ID) markers235are further visible inFIG.5.

The substrate205ofFIG.2Emay include or be coated with any previously-discussed substrate material and may be warped or shaped in any manner or include any devices discussed with respect to the substrate205ofFIG.2A, and therefore may be detected by, and be useful to calibrate a range sensor of the vehicle, such as a LIDAR, SONAR, SODAR, or radar sensor of the vehicle, and may be detected by a microphone or infrared sensor of the vehicle as well.

In some cases, the combined range sensor and camera calibration target250may have more or fewer apertures and corresponding visual markings than the four apertures and corresponding visual markings illustrated inFIG.2E.

Additional targets not depicted inFIG.2A-2Emay also be possible for calibration of different types of vehicle sensors. For example, targets for intrinsic or extrinsic calibration of infrared cameras or other types of infrared sensors of a vehicle102may include patterns formed using heating elements, optionally in front of, behind, or beside visual markings or substrate apertures or speakers for extrinsic calibration with cameras or range sensors or microphones, respectively. Targets for intrinsic or extrinsic calibration of microphones or other types of audio sensors of a vehicle102may include speakers or patterns formed using speakers, optionally in front of, behind, or beside visual markings or substrate apertures or heating elements for extrinsic calibration with cameras or range sensors or infrared sensors, respectively.

FIG.3illustrates a top-down view of a hallway calibration environment in which a vehicle traverses a drive path along which the vehicle is flanked by vehicle sensor calibration targets.

The hallway calibration environment300, which may also be referred to as a tunnel calibration environment, includes a thoroughfare305through which a vehicle102drives, the thoroughfare305flanked on either side by targets detectable by the sensors180of the vehicle102. The thoroughfare305may also be referred to as the drive path, the drive channel, the hallway, or the tunnel. Some of the targets are arranged in a left target channel310that is to the left of the vehicle102as the vehicle102traverses the thoroughfare305. Others of the targets are arranged in a right target channel315that is to the right of the vehicle102as the vehicle102traverses the thoroughfare305. InFIG.3, the targets in the left target channel310and right target channel315are all checkerboard camera targets200A as illustrated inFIG.2A, but they may include any other type of target discussed herein that is used to calibrate any vehicle sensor or combination of vehicle sensors. The left target channel310and right target channel315may include a combination of different target types, similarly to the calibration environment ofFIG.6; the targets need not all be of a uniform type as illustrated inFIG.3.

The vehicle102drives along the thoroughfare305, stopping after incremental amounts, for example, every foot, every N feet, every meter, or every N meters, where N is a number greater than zero, such as 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10. At each stop, the vehicle102captures data using each of its vehicle sensors, or at least each of the vehicle sensors that it intends to calibrate. The vehicle102stopping helps prevent issues caused by sensors running while the vehicle102is in motion, such as motion blur or rolling shutter issues in cameras. The vehicle102stopping also ensures that sensors can capture data while the vehicle102is in the same position, which may be important for extrinsic calibration of two or more sensors with respect to each other so that a location within data gathered by a first vehicle sensor (e.g., a range sensor such as a LIDAR or radar sensor) can be understood to correspond to a location within data gathered by a second vehicle sensor (e.g., a camera). The vehicle102may in some cases traverse the thoroughfare305multiple times, for example N times in each direction, where Nis, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, or 20.

The sensor targets illustrated inFIG.3are illustrated as each mounted on separate easel-style stands. Other types of stands are also possible, such as the type illustrated inFIG.4andFIG.5. Furthermore, multiple sensor targets, of the same type or of different types may be supported by each stand (as inFIG.4andFIG.5) even though this is not illustrated inFIG.3.

The sensor targets illustrated inFIG.3are illustrated such that some are positioned closer to the thoroughfare305while some are positioned farther from the thoroughfare305. Additionally, while some targets inFIG.3are facing a direction perpendicular to the thoroughfare305, others are angled up or down with respect to the direction perpendicular to the thoroughfare305. While the sensor targets illustrated inFIG.3all appear to be at the same height and all appear to not be rotated about an axis extending out from the surface of the target, it should be understood that the sensor targets may be positioned at different heights and may be rotated about an axis extending out from the surface of the target as in the targets ofFIG.4andFIG.5. Together, the distance from the thoroughfare305, the direction faced relative to the thoroughfare305, the clustering of targets, the height, and the rotation about an axis extending out from the surface of the target may all be varied and modified to provide better intrinsic and extrinsic calibration. That is, these variations assist in intrinsic calibration in that collection of data with representations of targets in various positions, rotations, and so forth ensures that targets are recognized as they should be by any sensor, even in unusual positions and rotations, and that any necessary corrections be performed to data captured by sensors after calibration. These variations assist in extrinsic calibration in that the different positions and rotations and so forth provide more interesting targets for range sensors, such as lidar, radar, sonar, or sodar, and allow range sensors to aid in interpretation of optical data collected by a camera of the vehicle102.

While the thoroughfare305of the hallway calibration environment300ofFIG.3is a straight path, in some cases it may be a curved path, and by extension the left target channel310and right target channel315may be curved to follow the path of the thoroughfare305.

While the hallway calibration environment300is effective in providing an environment with which to calibrate the sensors180of the vehicle102, it is inefficient in some ways. The hallway calibration environment300is not space efficient, as it occupies a lot of space. Such a hallway calibration environment300is best set up indoors so that lighting can be better controlled, so the hallway calibration environment300requires a large indoor space, and by extension, a lot of light sources, which is not energy-efficient or cost-efficient. Because of how much space the hallway calibration environment300takes up, it is more likely to have to be taken down and set back up again, affecting consistency of calibration between different vehicles whose sensors are calibrated at different times. Further, because the setup of the hallway calibration environment300requires the vehicle102to drive through it, different vehicles102might be aligned slightly differently in the thoroughfare102, and might drive a slightly different path through the thoroughfare102, and might stop at slightly different spots and/or frequencies along the drive, due to manufacturing differences in the vehicle102and due to human error in setting the vehicle102up, all of which affects consistency of the calibration. Trying to correct for all of these potential inconsistencies, and turning the vehicle around to move it through the hallway calibration environment300multiple times, is time and labor intensive, making the hallway calibration environment300time-inefficient. Additionally, because the targets are primarily to the left and right sides of the vehicle102hallway calibration environment300, vehicle sensors might not be as well calibrated in the regions to the front and rear of the vehicle. Using a thoroughfare305with some turns can help alleviate this, but again causes the hallway calibration environment300to take up more space, increasing space-inefficiency.

FIG.4illustrates a perspective view of a dynamic scene calibration environment in which a turntable that is at least partially surrounded by vehicle camera calibration targets rotates a vehicle so that the vehicle can perform intrinsic calibration of its camera sensors.

The dynamic scene calibration environment400ofFIG.4includes a motorized turntable405with a platform420that rotates about a base425. In some cases, the platform420may be raised above the floor/ground around the turntable405, with the base425gradually inclined up to enable the vehicle120to drive up the base425and onto the platform420, or to drive off of the platform420via the base425. A vehicle102drives onto the platform420of the turntable405, and the motors actuate to rotate platform420of the turntable405about the base425, and to thereby rotate the vehicle102relative to the base425(and therefore relative to the floor upon which the base425rests). While the arrows on the turntable405show a counter-clockwise rotation of the platform420, it should be understood that the platform420of the motorized turntable405can be actuated to rotate clockwise as well. The turntable405is at least partially surrounded by targets mounted on stands410. InFIG.4, the illustrated targets are all checkerboard-patterned camera calibration targets200A as depicted inFIG.2A, allowing for calibration of cameras of the vehicle102. In other embodiments, such as inFIG.5, the targets around the motorized turntable may include any other type of target discussed herein that is used to calibrate any vehicle sensor or combination of vehicle sensors.

In one embodiment, the platform420of the motorized turntable405may be rotated by predetermined intervals (measured in degrees/radians or an amount at a time), for example intervals of ten degrees, in between point the turntable stops so that the vehicle102can capture data with its sensors180. The platform420of the motorized turntable405can start and stop in this manner, and can eventually perform a full 360 degree rotation in this manner. The motorized turntable405may in some cases perform multiple full 360 degree rotations in one or both rotation directions (clockwise and counterclockwise), for example N rotations in each rotation direction, where Nis, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, or 20.

FIG.5illustrates a perspective view of a dynamic scene calibration environment in which a turntable that is at least partially surrounded by combined vehicle range sensor and vehicle camera calibration targets rotates a vehicle so that the vehicle can perform extrinsic calibration of its range sensors and camera sensors.

The dynamic scene calibration environment500ofFIG.5includes the same motorized turntable405as inFIG.4. A vehicle102drives onto the platform420of the turntable405, and the motors actuate to rotate the platform420of the turntable405, and thereby rotate the platform420(and the vehicle102) about the base425, clockwise, counter-clockwise, or one then the other. The turntable405is at least partially surrounded by targets mounted on stands410. InFIG.5, the illustrated targets include both a set of combined range/camera extrinsic calibration targets250as depicted inFIG.2Eand a set of trihedral radar calibration targets220as depicted inFIG.2D. Each stand410mounts two combined range/camera extrinsic calibration targets250and one trihedral radar calibration targets220, which in some cases may be a known distance from the combined range/camera extrinsic calibration targets250on the stand410, permitting extrinsic calibration between the radar and the range sensor (lidar, radar, sonar, sodar) and/or camera calibrated using the combined range/camera extrinsic calibration targets250.

As the vehicle102rotates about the base425on the platform420of the motorized turntable405, and/or during stops between rotations, the vehicle102and its computer110can detect the combined range/camera extrinsic calibration targets250using both its range sensors (e.g., lidar, etc.) and cameras by detecting the apertures225with the range sensors and the markings230around the apertures and the target identifier markings235with the cameras. In doing so, the vehicle102and its computer110can detect a center of the circular aperture225easily, since range sensors such as lidar typically provide a point cloud of depth measurements that can help identify where the widest pats of each circle are. The rings230detected by the camera will also have the same centers as the apertures, so the range sensor and camera know they are looking at the exact same locations for each of these center points. Thus, the camera and range sensor may be extrinsically calibrated so that their positional awareness of the surroundings of the vehicle102can be positionally aligned. The extrinsic calibration may, in some cases, output one or more matrices (e.g., one or more transformation matrices) used for transforming a camera location to a range sensor location or vice versa, via translation, rotation, or other transformations in 3D space. Calibration affects interpretation of data captured by the sensors after calibration is complete. The transformation(s) that are generated during this extrinsic calibration can include one or more types of transformations, including translations, stretching, squeezing, rotations, shearing, reflections, perspective distortion, distortion, orthogonal projection, perspective projection, curvature mapping, surface mapping, inversions, linear transformations, affine transformations, The translational and rotational transformations may include modifications to position, angle, roll, pitch, yaw, or combinations thereof. In some cases, specific distortions may be performed or undone, for example by removing distortion (e.g., parabolic distortion) caused by use of a specific type of lens in a camera or other sensor, such as a wide-angle lens or a fisheye lens or a macro lens.

The transformation(s) generated by the computer110of the vehicle102may be used for extrinsic calibration of a first sensor (e.g., the camera) with respect to a second sensor (e.g., LIDAR or RADAR or SONAR or SODAR or another range sensor), so that the computer102can map positions identified in the data output from each sensor to the real world environment around the vehicle102(e.g., in the field of view of the senbsors180of the vehicle102) and relative to each other, based on known relative positions of features identified within the outputs of each sensor. Such features may include the visual markings of the combined target250as identified by the camera, the apertures as identified by the range sensor, and optionally a trihedral target220affixed near or on the target250as in the environment500ofFIG.5. For example, if translation of positions in data captured by the first sensor to positions in the real world around the vehicle are already clear through intrinsic calibration, but translation of positions in data captured by the second sensor to positions in the real world around the vehicle are already clear through intrinsic calibration (or vice versa), then the transformation generated through this extrinsic calibration can translate positions in data captured by the second sensor to positions in the real world around the vehicle based on (1) the position in the real world around the vehicle of the data from the first sensor, and (2) the relative positioning of the position in the data from the first sensor and the position in the data from the second sensor. Thus, a sensor that has not been intrinsically calibrated can still be calibrated extrinsically relative to another sensor, and can still benefit from the increase in accuracy granted by the intrinsic calibration of that other sensor.

The trihedral targets220can also have a known distance from the combined range/camera extrinsic calibration targets250, and in some cases specifically from the centers of the apertures225and rings230of the targets250, allowing extrinsic calibration between the range sensor (e.g., radar) that recognizes the trihedral targets220and the range sensor (e.g., lidar) that recognizes the apertures225and the camera that recognizes the rings/markings230.

In other embodiments, the targets around the motorized turntable405may include any other type of target discussed herein that is used to calibrate any vehicle sensor or combination of vehicle sensors, such as the target200A ofFIG.2A, the target200B ofFIG.2B, the target200C ofFIG.2C, the target220ofFIG.2D, the target250ofFIG.2E, targets with heating elements detectable by infrared sensors of the vehicle102, targets with speakers detectable by microphones of the vehicle102, targets with reflective acoustic properties detectable by SONAR/SODAR/ultrasonic/infrasonic sensors of the vehicle102, or some combination thereof.

The stands410used inFIG.3-6may include any material discussed with respect to the substrate205, such as paper, cardboard, plastic, metal, foam, or some combination thereof. In some cases, certain stands may be made of a plastic such polyvinyl chloride (PVC) to avoid detection by certain types of range sensors, such as radar, which detect metal better than plastic.

In one embodiment, the platform420of the motorized turntable405may be rotated about the base425by predetermined intervals (measured in degrees/radians or an amount at a time), for example intervals of ten degrees, in between point the turntable stops so that the vehicle102can capture data with its sensors180. The intervals may be N degrees, where Nis, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, or 20. The platform420of the motorized turntable405can start and stop its rotation via activation and deactivation of its motor(s)730in this manner, and can eventually perform a full 360 degree rotation in this manner. The platform420of the motorized turntable405may in some cases perform multiple full 360 degree rotations about the base425in one or both rotation directions (clockwise and counterclockwise), for example N rotations in each rotation direction, where N is, for example, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, or 20.

FIG.6illustrates a top-down view of a dynamic scene calibration environment in which a turntable that is at least partially surrounded by both vehicle camera calibration targets and combined vehicle range sensor and vehicle camera calibration targets rotates a vehicle so that the vehicle can perform both intrinsic calibration of its camera sensors and extrinsic calibration of its range sensors and camera sensors.

The dynamic scene calibration environment600ofFIG.6is, in some ways, of a combination of the dynamic scene calibration environment400ofFIG.4and the dynamic scene calibration environment500ofFIG.5in that three types of targets are positioned around the motorized turntable405. These three types of targets include the checkerboard-patterned camera calibration targets200A used in the dynamic scene calibration environment400ofFIG.4and the trihedral radar calibration targets220and combined extrinsic calibration targets250used in the dynamic scene calibration environment500ofFIG.5. InFIG.6, all three types of targets stand separately from each other.

Any of the stands used inFIG.3-6may be any type of stands, such as easel-type stands, tripod-type stands, or the rod stands410with wide bases seen inFIG.4andFIG.5. Any of the stands used inFIG.3-6may include paper, cardboard, plastic, metal, foam, or some combination thereof as previously discussed. The stands in some cases may include motors and actuators enabling positions and/or angles of rotation of targets to be manipulated, for example to be more clearly visible given lighting conditions supplied by light sources620, or to access aa region of a point of view of a particular vehicle sensor from which that vehicle sensor has not captured enough data and therefore around which the vehicle sensor is under-calibrated. In some cases, these motors and their actuators may be controlled wirelessly by the vehicle102and/or scene surveying system610as discussed further with respect toFIG.7andFIG.14.

The dynamic scene calibration environment600ofFIG.6also includes a scene surveying system610, which may include one or more cameras, one or more range sensors (e.g., radar, lidar, sonar, sodar, laser rangefinder). This may take the form of a robotic total station (RTS). While the vehicle102, its sensors180, and the targets are enough to perform intrinsic calibration of sensors to correct for distortions, for example, and to perform extrinsic calibration of sensors to align locations within data captured by different sensors, for example, in some cases more information may be needed to understand how far these locations within data captured by different sensors are from the vehicle itself. The scene surveying system610captures visual and/or range data of at least a subset of the dynamic scene calibration environment600, including the motorized turntable410, at least some of the targets, and the vehicle102itself. Key points on the vehicle120may be tracked to identify the current pose (rotation orientation/position) of the vehicle102(and therefore of the platform420about the base425). Data captured by the scene surveying system610can also be sent to the vehicle102and used to verify the data captured by the sensors and the intrinsic and extrinsic calibrations performed based on this data. The dynamic scene calibration environment600also includes several light sources620set up around the turntable405. These light sources620may include any type of light source, such as incandescent bulbs, halogen bulbs, light emitting diodes (LEDs), and so forth. In some cases, the light sources620may include stage lights or spotlights. Mirrors and/or lenses may be used to manipulate light from the light sources620to relatively evenly illuminate the targets and/or the dynamic scene calibration environment600as a whole. Additional light sources may be positioned over the turntable405or behind some or all targets, especially if targets use a transparent or translucent substrate. The light sources620are used to improve readings from cameras of the vehicle and of the scene surveying system610, and can in some cases be controlled wirelessly by the vehicle102and/or scene surveying system610as discussed further with respect toFIG.7andFIG.14.

By including targets for intrinsic sensor calibration as well as targets for extrinsic calibration around the turntable405, the sensors180of the vehicle102can receive a comprehensive calibration in the dynamic scene calibration environment600. The dynamic scene calibration environments400,500, and600ofFIGS.4-6have several advantages over the static hallway calibration environment300ofFIG.3. First, the dynamic scene calibration environment is more space-efficient, as it only requires space for the vehicle turntable405, space for some targets, light sources620, and scene surveying system610. There is no need to clear space for a thoroughfare305or other long path. Because fewer light sources620are needed to light a smaller space like the dynamic scene calibration environment600than a larger space like the hallway calibration environment300, the dynamic scene calibration environment600is more energy-efficient in terms of lighting. Because the vehicle engine can be turned off after the vehicle is on the turntable405, the dynamic scene calibration environment600is more energy-efficient in terms of vehicular power usage. Because changing rotation directions is considerably quicker than performing a U-turn in a vehicle to change driving directions, and because a smaller space is more likely to remain untouched and therefore won't need take-down and setup after and before each use, the dynamic scene calibration environment600is more time-efficient. Because the target and lighting setup, and the movement of the vehicle102, can be better controlled, the calibration results are more consistent across vehicles102, which further allows calibration configurations of different vehicles102to be more readily compared and outliers identified as potentially symptomatic of a sensor defect or a miscalibration issue. Use of guide railings810as illustrated in and discussed with respect toFIG.8A-8Dfurther increases consistency of calibration by increasing consistency of positioning of the vehicle102along the turntable405. The dynamic scene calibration environment600also allows the sensors180of the vehicle102to capture data all around the vehicle102—including in the front and rear of the vehicle102, which receive less data in the hallway calibration environment300.

FIG.7illustrates a system architecture of an dynamic scene calibration environment.

The system architecture700of the dynamic scene calibration environment ofFIG.7includes a number of main elements, each with sub-components. The main elements include the autonomous vehicle102, a dynamic scene control bridge710, the motorized turntable system450, a lighting system760, a target control system770, a scene surveying system610, and a power supply system790.

The autonomous vehicle102includes the one or more sensors180, the one or more internal computing devices110, one or more wireless transceivers705(integrated with communication service116), and any other elements illustrated in and discussed with respect toFIG.1. The sensors180may in some cases include one or more GPRS receivers, Bluetooth® beacon-based positioning receivers, an inertial measurement unit (IMU), one or more cameras, one or more lidar sensors, one or more radar sensors, one or more sonar sensors, one or more sodar sensors, and/or one or more other sensors discussed herein, which may in some cases be used to identify when the vehicle102is not on the platform420, when the vehicle102is on the platform420, when the vehicle102in a defined position on the platform (e.g., as inFIG.8B), when the vehicle102have begun rotating from a stopped position, and/or when the platform420and/or the vehicle102have stopped rotating. The computing device110of the vehicle102(or its sensors180) may then automatically communicate one or more signals or messages through wired and/or wireless communication interfaces, for instance through the wireless transceiver705, to the bridge710, the computing device110of the vehicle102(if the communication is straight from the sensors180), and/or the computing device745of the motorized turntable system405to convey any of these observations/detections/identifications by the sensors180, which may be used to trigger various actions, such as rotation or stop of rotation of the turntable405, collection of sensor data or stop of collection of sensor data at the vehicle102, or some combination thereof.

The dynamic scene control bridge710includes one or more computing devices715and one or more wired and/or wireless transceivers720. The dynamic scene control bridge710is optional, but can serve as a “middleman” or “router” between the autonomous vehicle102and the remaining main elements of the system architecture700, such as the dynamic scene control bridge710, the motorized turntable system450, the lighting system760, the target control system770, the scene surveying system610, and the power supply system790. The dynamic scene control bridge710can in some cases convert file formats, perform mathematical operations such as operation conversions, or otherwise interpret instructions or data as necessary so that the vehicle120can successfully communicate with other elements. In fact, the dynamic scene control bridge710can perform similar conversions, mathematical operations, or interpretations in its role as a middleman between any two or more devices of the architecture700.

The motorized turntable system405includes a turntable structure725as well as one or more motors, encoders, actuators, and/or gearboxes730for actuating rotation of the turntable structure725while the vehicle102is on it. The motorized turntable725may include a platform420, which is a surface upon which the vehicle102rests during calibration. The platform420is rotatable about a base425of the motorized turntable structure725, with one or more motors730that, when actuated or activated, rotate the platform420about the base425, and which stop the rotation of the platform420about the base425when the motors730are deactivated. For example, the one or more motors730may rotate the platform420about the base425from a first rotational orientation to a second rotational orientation, or alternately back to the first rotational orientation (e.g., if the rotation is a 360 degree rotation or a multiple thereof). A rotational orientation of the platform420relative to the base425, or of the vehicle102relative to the base425, may alternately be referred to as a rotational position. The motorized turntable system405may include one or more sensors735, such as pressure sensors, for example to identify whether or not the vehicle102is on the turntable structure725, whether or not the vehicle102is positioned correctly on the turntable structure725, or how the vehicle's weight is distributed generally or across the platform420's top surface (which is in contact with the wheels and/or tires vehicle102) of the turntable structure725. The sensors735may in some cases identify when the platform420has no vehicle102on it, when the vehicle102is on the platform420, when the vehicle102in a defined position on the platform (e.g., as inFIG.8B), when the platform420and/or the vehicle102have begun rotating from a stopped position, and/or when the platform420and/or the vehicle102have stopped rotating. The motorized turntable system405may then automatically communicate one or more signals or messages through wired and/or wireless communication interfaces to the bridge710, vehicle102, and/or the computing device745of the motorized turntable system405to convey any of these observations/detections/identifications by the sensors735, which may be used to trigger various actions, such as rotation or stop of rotation of the turntable405, collection of sensor data or stop of collection of sensor data at the vehicle102, or some combination thereof. The controller740may be used to control the actuation of the motors, encoders, actuators, and/or gearbox(es)730, for example to control a speed or rate or angular velocity of rotation, an angular acceleration (or deceleration) of rotation, a direction of rotation (e.g., clockwise or counterclockwise), or some combination thereof. The motorized turntable system405includes one or more computing devices745and one or more wired and/or wireless transceivers750, through which it may interact with the vehicle102, the dynamic scene control bridge710, or any other element in the architecture700.

The lighting system760includes one or more light sources620and one or more motors and/or actuators762for activating or turning on each of the light sources620, disabling or turning off each of the light sources620, fading or dimming each of the light sources620, brightening each of the light sources, or moving each of the light sources620with an actuated motor (e.g., to shine on a particular target). The lighting system760includes one or more computing devices764and one or more wired and/or wireless transceivers766, through which it may interact with the vehicle102, the dynamic scene control bridge710, or any other element in the architecture700.

The target control system770includes one or more targets and target support structure772. The targets may include one or more of any of the targets200A,200B,200C,220, and/or250illustrated inFIG.2A-2E, any other target described herein, any other sensor calibration target, or some combination thereof. The target support structures may include easel-type support structures, tripod-type support structures, or the rod-type support structures410with wide bases seen inFIG.4andFIG.5. The target support structures may include any material discussed with respect to the substrate205, such as paper, cardboard, plastic, metal, foam, or some combination thereof. In some cases, certain stands may be made of a plastic such polyvinyl chloride (PVC) to avoid detection by certain types of range sensors, such as radar, which detect metal better than plastic.

The targets and/or support structures720may in some cases be motorized, and as such, the target control system770may include motors and actuators774that it can use to move the targets, for example as requested by the vehicle102to optimize calibration. For example, the target support structures may include a robotic arm with ball joints and/or hinge joints that may be actuated using the motors and actuators774to translate a target in 3D space and/or to rotate a target about any axis. The motors and actuators773may alternately only control a single type of movement for a particular target, for example by enabling a target to rotate about the rod of a stand410. The target support structure772may also include wheels or legs, which may be actuated by the motors774to enable the entire target support structure772to move, and with it, the target(s) it supports. The target control system770includes one or more computing devices776and one or more wired and/or wireless transceivers778, through which it may interact with the vehicle102, the dynamic scene control bridge710, or any other element in the architecture700.

The scene surveying system610includes a surveying device support structure780, such as a tripod or any other structure discussed with respect to the target support structure772, and one or more sensors782coupled to the support structure780. The sensors782of the scene surveying system610, like the sensors180of the vehicle102, may include one or more cameras of any type (e.g., wide-angle lens, fisheye lens), one or more range sensors (e.g., radar, lidar, emdar, laser rangefinder, sonar, sodar), one or more infrared sensors, one or more microphones, or some combination thereof. Using these, the scene surveying system610can capture a representation of the entire dynamic scene, including the vehicle102, allowing determination of distances between the vehicle102and various targets. In some cases, either the vehicle102or the scene surveying system610or both may request adjustment of lighting through the lighting system760and/or adjustment of target positioning via the target control system770. The scene surveying system610includes one or more computing devices784and one or more wired and/or wireless transceivers784, through which it may interact with the vehicle102, the dynamic scene control bridge710, or any other element in the architecture700. In some cases, feature tracking and/or image recognition techniques applied using the computing device784may be used with the a camera and/or the radar, lidar, sonar, sodar, laser rangefinder, and/or other sensors782of the scene surveying system610to identify when the platform420has no vehicle102on it, when the vehicle102is on the platform420, when the vehicle102in a defined position on the platform (e.g., as inFIG.8B), when the platform420and/or the vehicle102have begun rotating from a stopped position, and/or when the platform420and/or the vehicle102have stopped rotating. The scene surveying system610may then automatically communicate one or more signals or messages through wired and/or wireless communication interfaces to the bridge710, vehicle102, and/or motorized turntable system405to convey any of these observations/detections/identifications by the scene surveying system610, which may be used to trigger various actions, such as rotation or stop of rotation of the turntable405, collection of sensor data or stop of collection of sensor data at the vehicle102, or some combination thereof. In some cases, the scene surveying system610may simply be referred to as a camera or as another sensor that the scene surveying system610includes.

The power supply system790may include batteries, generators, or may plug into an outlet and into the power grid. The power supply system790may supply power to the various elements and components of the system architecture700, including at least the dynamic scene control bridge710, the motorized turntable system450, the lighting system760, the target control system770, and the scene surveying system610. The power supply system790may also charge the vehicle102before, during, and/or after calibration, for example if the vehicle102is electric or hybrid. The power supply system790may also intelligently scale voltage, amperage, and current as appropriate for each element and component of the system architecture700, and to do so it may include a computing system1500(not pictured). It may also include a wired and/or wireless transceiver (not pictured) through which it may interact with the vehicle102, the dynamic scene control bridge710, or any other element in the architecture700.

The computing devices110,715,745,764,776, and784may each, at least in some cases, include at least one computing system1500as illustrated in or discussed with respect toFIG.15, or may include at least a subset of the components illustrated inFIG.15or discussed with respect to computing system1500.

FIG.8Aillustrates a top-down view of a turntable with a vehicle guide rail, and a vehicle driving onto the turntable centered relative to the vehicle guide rail.

The motorized turntable system405ofFIG.8Aincludes a guide railing810on the platform420of the turntable405, particularly on a top surface of the platform420, so that the guide railing810rotates along with the rest of the platform420about the base425when the turntable motor730is activated. The guide railing810includes two substantially parallel straight rails840with a space in between, illustrated inFIG.8Aas parallel vertically-oriented lines, the space between the two straight rails840corresponding to a distance between two wheels and/or tires of the vehicle102(e.g., left and right front wheels/tires of the vehicle102and/or left and right rear wheels/tires) in such a way that the vehicle102can straddle the two straight rails840with at least one left wheel and/or tire on a left side of the left straight rail840and at least one right wheel and/or tire on a right side of the right straight rail840. The rails of the guide railing810—that is, the two straight rails840and the two slanted rails845—may extend from the top surface of the platform420in a direction perpendicular to the top surface of the platform420.

At one end, these parallel rails end in a “stop” member850, which may include a rail, a wall, a bump, or an inclined ramp, the stop member850gently stopping the vehicle102from continuing further forward once the vehicle has reached a defined position astride the two straight rails840. That is, each stop member850may include a vertical wall or rail that extends in a direction perpendicular to the top surface of the platform420, or an inclined ramp or bump that gradually increases in height (height being a direction perpendicular to the surface of the platform420) along a direction in which the vehicle102is driving onto or along the platform420to form an incline that, if the vehicle102were to drive at least partially onto the incline, would push the vehicle102backward. The incline may be straight, have a convex curve, or have a concave curve. The incline may be part of a bump that goes up in height and then back down, such as a speed bump. That is, regardless of the form of the stop members850, if the vehicle102reaches the defined position at which it is pictured inFIG.8Band attempts to continue driving toward the one or more stop members850, the vehicle102will come into contact with one or more stop members850, which will push the vehicle back into the defined position, either gently using an incline or more forcefully using a wall or rail. The vehicle102may drive forwards onto the platform420and toward the stop members850as illustrated inFIG.8A(i.e., with the front side of the vehicle102facing the direction of driving and the stop members850), or may reverse onto the platform420and toward the stop members850(e.g., drive with the rear side of the vehicle102facing the direction of driving and the stop members850). A wall or rail included in a stop member850may in some cases be padded using foam, silicone, rubber, or another soft or bouncy material that cushions the force applied by the stop member850on the vehicle102and the force applied by the vehicle102on the stop member850. The stop members850of the guide railing810ofFIG.8Aare illustrated as two relatively short horizontal lines (perpendicular to the parallel vertical guide rails) at the top endpoints (i.e., posterior ends) of the parallel straight rails840of the guide railing810. In some cases, one of the two stop members850may be missing, as stopping one tire/wheel from progressing may be sufficient to stop the vehicle102from moving beyond the stop member850. While the two stop members850are illustrated as a separate member for each of the two straight rails840, in some cases (not pictured), a single longer stop member850can be used that bridges the gap between the straight rails840and comes into contact with the posterior ends of both straight rails840in a direction that is perpendicular to both straight rails, extending beyond the posterior ends of one or both straight rails840.

The other end (i.e., the anterior end) of each of the two parallel straight rails840ends in a slanted rail845. Relative to the two parallel straight rails840, the two slanted rails845slant towards one another. The slanted rails845are generally not parallel to one another or to the straight rails840, though may include portions that are parallel to one or both (e.g., when the slanted rails845are curved). That is, the leftmost slanted rail845slants to the right as it proceeds in further downward (i.e., in a more anterior direction), while the rightmost slanted rail845slants to the left as it proceeds in further downward (i.e., in a more anterior direction). In the guidance rail810illustrated inFIG.8A, the two slanted rails845are straight as well, and approach a single point (i.e., a vertex) at which they meet/converge. That point at which the slanted rails845meet/converge may be along a center line805/815that is parallel to the two straight rails840and that is centered (i.e., equidistant) between the two straight rails840. Accordingly, the two slanted rails845, together, form a “V” shape. In some cases (not illustrated), the two slanted rails845need not meet at the point they both approach, thereby remaining separate and effectively removing a bottom portion/fraction of the “V” shape that includes the vertex. In some cases (not illustrated), the vertex of the “V,” the vertices between the slanted rails845and the straight rails840, and/or any other vertices formed by the slanted rails845and/or the straight rails840may be smoothed out so as not to present a risk of puncturing or otherwise damaging a tire or wheel of the vehicle102. In some cases (not illustrated), the two slanted rails845are curved rather than straight, for example forming a “U” shape rather than a “V” shape, or at least a top portion of a “U” shape. That is, each of the slanted rails845may include one or more curves that each form a portion of a circle or a portion of an ellipse, each curve of which may for example be concave facing inward (i.e., facing the space between the two parallel straight rails840) and convex facing outward (i.e., facing the base425of the turntable and the rest of the calibration environment) or vice versa. In some cases, each of the slanted rails840may include one or more straight portions and one or more curved portions, and may for example form a curve similar to a curve formed by graphing a tangent function (e.g., ƒ*tan (x) where ƒ is a positive or negative constant) in a 2-dimensional X-Y space.

For structural integrity, the guide railing810may also include optional braces (not pictured) connecting the left and right straight rails840to one another through the space in between the two straight rails840. Similar braces may exist between the two slanted rails845, connecting the two slanted rails845. Similar braces may exist between the two stop members850, connecting the two stop members850. The braces may include braces that are perpendicular to the parallel vertical guide rails or diagonal relative to the parallel vertical guide rails (e.g., forming one or more cross (“X”) structures). The braces may include braces that are perpendicular to at least a portion of one or both of the slanted rails845or diagonal relative to at least a portion of one or both of the slanted rails845.

The vehicle102ofFIG.8Aapproaches the motorized turntable system405along a centered path805, represented by dashed line arrow, that is centered with respect to the turntable405and the guide railing810. As a result, the vehicle102will successfully drive onto the center of the platform420of the motorized turntable405, eventually reaching a defined position in which the wheels and/or tires of the vehicle102straddle (i.e., are positioned astride) the two straight rails840and abut the stop members850, the vehicle102illustrated in this defined position inFIG.8B. The defined position represents the position that the guide railing810guides the vehicle102into.

In some cases (not pictured), the guide railing810may intentionally be positioned off-center (horizontally and/or vertically) relative to the center of the platform420of the motorized turntable system405. For example, the guide railing810may be moved further forward relative to the platform420to accommodate and/or center a larger or longer vehicle102than the illustrated sedan-style vehicle102(e.g., a van, truck, or limousine) on the platform420, or further backward relative to the platform420to accommodate and/or center a smaller or shorter vehicle102than the illustrated sedan-style vehicle102(e.g., a compact automobile, buggy, or all-terrain vehicle) on the platform420. The guide railing810may be moved left or right as well, as it may be desirable for the vehicle102to move translationally in space during certain types of sensor calibrations rather than to just rotate about an axis that passes through the vehicle, effectively rotating in small circular “laps” within a radius of the platform420but with the benefit of the precision motor control of the turntable405(as opposed to using the vehicle102's engine for propulsion) and of not having to consume fuel or battery power of the vehicle102. In some cases, the guide railing810may be adjusted dynamically. To achieve such adjustability in a dynamic fashion, the guide railing810itself may be attached to the platform420via additional rail tracks (not shown) along the platform420that are optionally recessed into the platform, the rail tracks along the platform420allowing the guide railing810to be slid along the rail tracks relative to the surface of the platform420, with one or more latches and/or magnets and/or screws used to lock the guide railing810in a particular position along the rail tracks along the platform420. Movement of the guide railing810along such rail tracks along the platform420may also be activated and/or deactivated using one or more motors that may be controlled by the motorized turntable system405and/or by the vehicle102and/or otherwise as discussed with respect to actuation of the motors, encoders, actuators, and/or gearbox(es)730.

FIG.8Billustrates a vehicle having successfully driven onto the turntable centered along the vehicle guide rail, the vehicle thus centered with respect to the turntable.

The vehicle102inFIG.8B, which is illustrated as transparent with a dashed line outline and grey-filled wheels/tires, has driven onto the motorized turntable system405. Through the transparent vehicle outline, we can see that the wheels and tires of the vehicle102are adjacent to the outsides of the parallel rails of the guide railing810, and the stop member850has stopped the vehicle102at the defined position, which inFIG.8Bis in the center of the platform of the turntable405, but may alternately be positioned further forward, backward, and/or to one of the sides of the platform420as discussed with respect toFIG.8A.

While the two straight rails840illustrated inFIGS.8A-8Dare only long enough for two wheels and/or tires of the vehicle102(either the front left and right wheels/tires or the rear left and right wheels/tires) to stand astride the two straight rails840while the vehicle102is in the defined position, in some cases the two straight rails840may be long enough so that all four or more wheels and/or tires of a vehicle102stand astride the two straight rails840while the vehicle102is in the defined position. That is, if the vehicle is a six-wheeler or eight-wheeler or ten-wheeler truck or other long vehicle102, all of the left and right pairs of wheels, or any subset of the pairs of wheels, can stand astride the two straight rails840while the vehicle102is in the defined position.FIG.8Cillustrates a top-down view of a turntable with a vehicle guide rail, and a vehicle driving onto the turntable off-center relative to the vehicle guide rail.

The vehicle102ofFIG.8Capproaches the platform of the motorized turntable system405along an un-centered path820, represented by dashed line arrow, which is off-center with respect to the turntable405and the guide railing810, as is visible when compared to the center line815, represented by a dotted line. The center line815is a line representing the centered path805fromFIG.8Aand thus represents center with respect to the platform of the turntable405and the guide railing810and the defined position of the vehicle102inFIG.8B. The center line815is illustrated inFIG.8Cto visibly highlight that the un-centered path820is off-center with respect to the platform of the turntable405and the guide railing810and the defined position of the vehicle102inFIG.8B. As a result, the vehicle102will drive onto the platform of the motorized turntable405off-center with respect to the platform of the turntable405and the guide railing810and the defined position of the vehicle102inFIG.8B, and will approach the guide railing810off-center off-center with respect to the platform of the turntable405and the guide railing810and the defined position of the vehicle102inFIG.8Bas visible inFIG.8D.

FIG.8Dillustrates a vehicle having driven partially onto the turntable while off-center relative to the vehicle guide rail, the vehicle guide rail guiding the vehicle's path to center the vehicle with respect to the turntable.

The vehicle102inFIG.8D, which is illustrated as transparent with a dashed line outline and grey wheels/tires, has driven onto the motorized turntable system405off-center. Through the transparent vehicle outline, we can see that one of the wheels/tires of the vehicle102is contacting the slanted rails845(e.g., the “V” or “U” portion) of the guide railing810at a guidance point830. If the vehicle102keeps driving forward relative to the position it is illustrated in inFIG.8D, the the slanted rail840will push back against the vehicle102at the guidance point830(i.e., the contact point), as will further contact points along the same slanted rail840of the guide railing810as the vehicle102continues to progress forward (diagonally) pressing against the same slanted rail840, and thus the slanted rail840of the guide railing810will automatically guide the vehicle102to the right toward the centered/defined position until the vehicle102reaches the substantially parallel vertical rail portions of the guide railing810, after which the vehicle102can drive straight without encumbrance. Thus, through application of forward torque/force by the vehicle102on the slanted rail845at the guidance point830, and through reciprocal force pushing back diagonally on the vehicle102from the guidance point830of the slanted rail420of the guide railing810, eventually, the slanted rail420of the guide railing810will assist the vehicle102into a centered position with respect to the guide railing810and the platform420of the turntable405, and the vehicle102will ultimately reach the defined position that the vehicle102is illustrated having reached inFIG.8Bafter moving forward along the straight rails840. WhileFIGS.8C and8Dillustrate the effect of the guide railing810on centering the vehicle810along the platform420of the turntable system405when the vehicle102is approaching off-center to the left relative to the guide railing810by pushing the vehicle102to the right, a similar effect occurs when the vehicle102is approaching off-center to the right relative to the guide railing810(not illustrated) by pushing the vehicle102to the left. Thus, the guide railing810allows for easy and consistent positioning of the vehicle102at a defined position along the platform420of the turntable405.

FIG.9is a flow diagram illustrating operation of a calibration environment.

At step905, a high resolution map of calibration environment is generated. This may be performed using the scene surveying system610, for example.

At step910, all sensors180on the vehicle102are run in the calibration environment, for example at different rotation positions of the vehicle102, which is rotated using motorized turntable405. At step915, the vehicle102generates a calibration scene based on its sensors180, based on (a) synchronized sensor data, (b) initial calibration information, (c) vehicle pose information, and (d) target locations.

At step915, the calibration systems in the vehicle read the calibration scene and: (a) detect targets in each sensor frame, (b) associate detected targets, (c) generate residuals, (d) solve calibration optimization problem, (e) validate calibration optimization solution, and (f) output calibration results. At step925, the calibration results are tested against acceptable bounds and checked for numerical sensitivity. Successful calibration measurements are stored and logged, along with a minimal subset of data needed to reproduce them

FIG.10is a flow diagram illustrating operations for intrinsic calibration of a vehicle sensor using a dynamic scene.

At step1005, a vehicle102is rotated into a plurality of vehicle positions over a course of a calibration time period using a motorized turntable405. The vehicle102and motorized turntable405are located in a calibration environment. At step1010, the vehicle102captures a plurality of sensor capture datasets via a sensor coupled to the vehicle over the course of the calibration time period by capturing at least one of the plurality of sensor capture datasets while the vehicle is at each of the plurality of vehicle positions.

At step1015, an internal computing system110of the vehicle102receives the plurality of sensor capture datasets from the sensor coupled to the vehicle over a course of a calibration time period. At step1020, the internal computing system110of the vehicle102identifies, in the plurality of sensor capture datasets, one or more representations of (at least portions of) the calibration environment that include representations of a plurality of sensor targets. The plurality of sensor targets are located at known (i.e., previously stored) positions in the calibration environment. At steps1025-1030, the sensor is calibrated based on the representations of a plurality of sensor targets identified in the plurality of sensor capture datasets.

More specifically, at step1025, the internal computing system110of the vehicle102identifies positions of the representations of the plurality of sensor targets within the one or more representations of (at least portions of) the calibration environment. If the sensor being calibrated is a camera, and the one or more representations of (portions of) the calibration environment are images, then the representations of the sensor targets may be areas within the one or more images comprised of multiple pixels, which the computing system110of the vehicle102can identify within the one or more images by generating high-contrast versions of the one or more images (i.e., “edge” images) that are optionally filtered to emphasize edges within the image, and by identifying features within the image comprised of one or more of those edges, the features recognizable as portions of the target. For example, the vertices and/or boxes in the checkerboard pattern210A or the ArUco pattern210B, curves or vertices in the crosshair pattern210C, the circular ring marking patterns230, or combinations thereof, may each be visually recognized as features in this way. Similarly, if the sensor being calibrated is a radar sensor, the radar sensor may recognize the trihedral shape215of the target220as a feature due to its reflective pattern that results in a high radar cross section (RCS) return. Similarly, if the sensor being calibrated is a lidar sensor, the lidar sensor may recognize the surface of the substrate205of the target250and the apertures225/240within the substrate205of the target250, which may be recognized as a feature due to the sharp changes in range/depth at the aperture.

At step1030, the internal computing system110of the vehicle102generates a transformation that maps (1) the positions of the representations of the plurality of sensor targets within one or more representations of (portions of) the calibration environment to (2) the known positions of the plurality of sensor targets within the calibration environment. Other information about the plurality of sensor targets, such as information storing visual patterns or aperture patterns of the sensor targets, may also be used to generate the transformation. For example, if the sensor being calibrated is a camera, and the computing device110knows that an image should have a checkerboard pattern210A of a sensor target200A, and recognizes a warped or distorted variant of the checkerboard pattern210A (e.g., because the camera includes a fisheye lens or wide-angle lens), then the computing device110may use its knowledge of the way that the checkerboard should look, such as how far the vertices are from each other, that they should form squares, and that the squares are arranged in a grid pattern—to generate a transformation that undoes the distortion caused by the camera, thereby mapping the vertices detected in the image to real-world positions, at least relative to one another. In other words, the transformation includes one or more projective transformations of various 2-D image coordinates of sensor target features into 3-D coordinates in the real world and optionally back into 2-D image coordinates that have been corrected to remove distortion and/or other sensor issues.

Because the computing device110knows ahead of time exactly where the sensor targets are in the calibration environment, the transformation may also map the positions of the vertices in the image (and therefore the positions of the representations of the sensor targets in the representation of the calibration environment) to real-world positions in the calibration environment. The transformation(s) that are generated during intrinsic sensor calibration at step1030can include one or more types of transformations, including translations, stretching, squeezing, rotations, shearing, reflections, perspective distortion, distortion, orthogonal projection, perspective projection, curvature mapping, surface mapping, inversions, linear transformations, affine transformations, The translational and rotational transformations may include modifications to position, angle, roll, pitch, yaw, or combinations thereof. In some cases, specific distortions may be performed or undone, for example by removing distortion caused by use of a specific type of lens in a camera or other sensor, such as a wide-angle lens or a fisheye lens or a macro lens.

Step1030may be followed by step1005and/or by step1010if calibration is not yet complete, leading to gathering of more sensor capture datasets and further refinement of the transformation generated at step1030. Step1030may alternately be followed by step1045.

The previously stored information about the plurality of sensor targets may be from a high-definition map generated as in step905ofFIG.9, may be from a second sensor on the vehicle, or may simply be based on a prior understanding of the sensor targets. For example, the internal computing system110of the vehicle102understands what a checkerboard pattern210A is and that the grid it forms ought to look include a grid of parallel and perpendicular lines under normal conditions. Because of this, the internal computing system110understands that if the representation it received from the sensor (camera) of a target with a checkerboard pattern210A forms a grid warped or distorted by a wide-angle lens or fisheye lens, this difference (step1030) can be corrected by the internal computing system110by correctively distorting or warping the image captured by the sensor (camera) by to reverse the warping or distortion in the representation until the corrected checkerboard looks like it should. This corrective warping or distortion is the correction generated in step1035. The correction may also include a translation along X, Y, or Z dimensions, a rotation along any axis, a warp or distortion filter, a different type of filter, or some combination thereof

Steps1045-1060concern operations that occur after calibration is complete (i.e., post-calibration operations). At step1045, the sensor of the vehicle captures a post-calibration sensor capture dataset after the calibration time period, after generating the transformation, and while the vehicle is in a second position that is not in the calibration environment. At step1050, the computing device110of the vehicle102identifies a representation of an object within a representation of a scene identified within the post-calibration sensor capture dataset. At step1055, the computing device110of the vehicle102identifies a position of the representation of the object within the representation of the scene. At step1060, the computing device110of the vehicle102identifies a position of the object relative to the second position of the vehicle by applying the transformation to the position of the representation of the object within the representation of the scene.

Note that capture of data by the sensors180of the vehicle102may occur in parallel with calibration of the sensors180of the vehicle102. While an initial correction is generated at step1035, the vehicle102may continue to rotate, and its sensors180may continue to capture more sensor data, hence the dashed lines extending back up to steps1005and1010from step1035. When step1035is reached a second, third, or Nth time (where N is any integer over 1), the correction generated the first time step1035was reached may be updated, revised, and/or re-generated based on the newly captured sensor data when step1035is reached again. Thus, the correction becomes more accurate as calibration continues.

For some additional context on intrinsic calibration: LIDAR intrinsic properties may include elevation, azimuth, and intensity. Camera intrinsic properties may be given as matrices based on camera region/bin, and may track projection, distortion, and rectification. All sensors' intrinsic properties (including LIDAR and camera) may include position in X, Y, and/or Z dimensions, as well as roll, pitch, and/or yaw.

FIG.11is a flow diagram illustrating operations for extrinsic calibration of two sensors in relation to each other using a dynamic scene.

At step1105, a vehicle102is rotated into a plurality of vehicle positions over a course of a calibration time period using a motorized turntable405. At step1110, the vehicle102captures a first plurality of sensor capture datasets via a first sensor coupled to the vehicle over the course of the calibration time period by capturing at least one of the first plurality of sensor capture datasets while the vehicle is at each of the plurality of vehicle positions. At step1115, the vehicle102captures a second plurality of sensor capture datasets via a second sensor coupled to the vehicle over the course of the calibration time period by capturing at least one of the first plurality of sensor capture datasets while the vehicle is at each of the plurality of vehicle positions. Either of steps1110and1115can occur first, or they can occur at least partially in parallel.

At step1120, the internal computing system110of the vehicle102receives the first plurality of sensor capture datasets from the first sensor and the second plurality of sensor capture datasets from the second sensor. At step1125, the internal computing system110of the vehicle102identifies, in the first plurality of sensor capture datasets, representations of a first plurality of sensor target features, the first plurality of sensor target features detectable by the first sensor due to a type of the first plurality of sensor target features being detectable by sensors of a type of the first sensor. At step1130, the internal computing system110of the vehicle102identifies, in the second plurality of sensor capture datasets, representations of a second plurality of sensor target features, the second plurality of sensor target features detectable by the second sensor due to a type of the second plurality of sensor target features being detectable by sensors of a type of the second sensor. Either of steps1125and1130can occur first, or they can occur at least partially in parallel.

The first plurality of sensor target features and the second plurality of sensor target features may be on the same targets; for example, if the first sensor is a camera, and the second sensor is a LIDAR sensor, and plurality of sensor targets are the combined extrinsic calibration targets250ofFIGS.2E and5, then the first plurality of sensor target features may be the visual markings (rings)230detectable by the camera, while the second plurality of sensor target features are the apertures225detectable by the LIDAR. Alternately, the plurality of sensor target features may be on different targets; for example, the first sensor may be a radar sensor and the first plurality of sensor target features may be the trihedral radar calibration targets220, while the second sensor may be any other sensor (e.g., camera, lidar) and the second plurality of sensor target features may be, for example, the visual markings (rings)230or apertures225of the combined extrinsic calibration targets250, or a pattern210of a camera intrinsic target such as the targets200A-C, or any other target described herein.

At step1135, the internal computing system110of the vehicle102compares the relative positioning of the representations of the first plurality of sensor target features and the representations of the second plurality of sensor target features to known relative positioning of the first plurality of sensor target features and the second plurality of sensor target features. In some cases, the relative positioning may be determined based on comparison of a position of a particular point in one representation, such as the center, to a particular point in the another representation to which it is being compared, such as the center. Points that can be used instead of the center may include or the highest point, lowest point, leftmost point, rightmost point, a point that is centered along one axis but not another, a point at the widest portion of the representation, a point at the narrowest portion of the representation, a point at a particular edge or vertex, or some combination thereof. At step1140, the internal computing system110of the vehicle102generates a transformation based on the comparison, such that the transformation aligns a first location identified by the first sensor and a second location identified by the second sensor.

As a first example, the first sensor may be a camera and the second sensor may be a LIDAR sensor, and the first plurality of sensor target features and the second plurality of sensor target features may both be features of the combined extrinsic calibration targets250ofFIGS.2E and5such that the first plurality of sensor target features are the visual markings (rings)230detectable by the camera and the second plurality of sensor target features are the apertures225detectable by the LIDAR. In such a case, the internal computing system110of the vehicle102identifies a center of a particular aperture225based on the LIDAR data, and identifies a center of a ring230based on the camera data, compares these at step1135and identifies a relative distance between the two locations based on the internal computing system110's current geographic understanding of the calibration environment. Because the internal computing system110understands that these two points should represent the same location in the real world (i.e., their relative positioning indicates no distance between them), the internal computing system110generates a transformation-which may include, for example, a translation along X, Y, and/or Z dimensions, a rotation along any axis, a warp or distortion filter, or some combination thereof—that aligns these location points. That is, the transformation translates (1) a mapping of a point from the one sensor's capture data set to a real world position into (2) a mapping of a point from the other sensor's capture data set to the same real world position. While, with just a pair or two of such points, there may be multiple possible transformations that can perform this alignment, the internal computing system110can generate a transformation that works consistently for an increasing number of pairs such sets of points—for example, for each aperture225and ring230combinations of a target250, and for each target250in the calibration environment. As the number of pairs increases, the number of possible transformations that can successfully align these. Different sensors may map the world around them differently; for example, if the camera includes a wide-angle lens while the other sensor (e.g., LIDAR) does not include an analogous distortion effect, the transformation may include some radial movement or other compensation for distortion.

As a second example, the first sensor may be a radar sensor and the second sensor may be a LIDAR sensor, and the first plurality of sensor target features may be trihedral radar calibration targets220while the second plurality of sensor target features may be apertures225of a combined target250or the planar boundaries of a substrate205of a camera target200, each of which is a known distance away from the nearest trihedral radar calibration targets220. In such a case, the internal computing system110of the vehicle102identifies a location of the trihedral radar calibration targets220based on radar sensor data and a location of the LIDAR target feature based on LIDAR sensor data, compares these at step1135and identifies a relative distance between the two locations based on the internal computing system110's current geographic understanding of the calibration environment. Because the internal computing system110understands that these two points should be a known distance away in a particular direction at a particular angle in the real world, the internal computing system110generates a transformation-which may include, for example, a translation along X, Y, and/or Z dimensions, a rotation along any axis, a warp or distortion filter, or some combination thereof—that aligns these location points to match the same known distance away in the particular direction at the particular angle as in the real world. While initially there may be multiple possible transformation that can perform this, the internal computing system110can generate a transformation that works consistently for multiple such sets of points—for example, for each trihedral radar calibration target220and each nearby LIDAR target feature pair in the calibration environment.

At step1145, the internal computing system110of the vehicle102receives, from the first sensor and second sensor, post-calibration sensor capture datasets captured by the first sensor and second sensor after the calibration time period. At step1150, the internal computing system110of the vehicle102applies the transformation generated in step1140to one or both of the post-calibration sensor capture datasets. For example, a representation of a particular object can be identified in a post-calibration sensor capture dataset captured by one sensor after calibration, and the transformation can be applied to find the same object within another post-calibration sensor capture dataset captured by another sensor after calibration. A real-world position of the same object may be found relative to the vehicle102based on intrinsic calibration of at least one of the two sensors and/or based on the transformation. In some cases, a representation of an entire space—that is, a three-dimensional volume—in one post-calibration sensor capture dataset captured by one sensor after calibration may then be identified in another post-calibration sensor capture dataset captured by another sensor by applying the transformation to multiple points within the space. Important points, such as vertices (e.g., corners of a room), edges (e.g., edges of a room), or other features may be selected as at least some of these points. With two aligned representations of a 3-D space, objects can be identified around the vehicle that might not otherwise be. For example, a pedestrian wearing all black might not visually stand out against (e.g., contrast against) a background of an asphalt road at night, but a RADAR or LIDAR might easily identify the pedestrian, and the transformation will still allow the computer110of the vehicle102to understand where that pedestrian is in its camera footage, allowing the vehicle to pay close attention to visual cues from the pedestrian that the RADAR or LIDAR might not catch or understand, such as presence or lack of a pet or small child accompanying the pedestrian. Developing the vehicle's understanding of its surroundings by aligning real-world (and relative) mappings of the inputs it receives from its sensors can save lives in the field of autonomous vehicles by allowing the best aspects of multiple sensors to complement one another to develop a comprehensive view of the vehicle's surroundings. No sensor is perfect at detecting everything-range sensors can see range/depth but not color or brightness, and can have trouble seeing small or fast-moving objects—while cameras can see color and brightness and visual features but can have trouble with depth perception. Thus, each sensor has its strengths, and the alignment made possible by the extrinsic calibration processes discussed inFIG.11can allow the best aspects of each sensor (the “pros” of each sensor) to complement each other and compensate for the downsides of each sensor (the “cons” of each sensor). Note that capture of data by the sensors180of the vehicle102may occur in parallel with calibration of the sensors180of the vehicle102. While an initial transformation is generated at step1140, the vehicle112may continue to rotate, and its sensors180may continue to capture more sensor data, hence the dashed lines extending back up to steps1105and1110and1115from step1140. When step1140is reached a second, third, or Nth time (where N is any integer over 1), the transformation generated the first time step1140was reached may be updated, revised, and/or re-generated based on the newly captured sensor data when step1140is reached again. Thus, the transformation becomes more accurate as calibration continues.

For some additional context on extrinsic calibration, all sensors' extrinsic properties may include relative positions in X, Y, and/or Z dimensions, as well as roll, pitch, and/or yaw. Target and vehicle locations are ground truthed via the mapping system discussed in step910and further as discussed with respect to the transformation of step1030ofFIG.10. Sensors of the vehicle102and scene surveying system610record the scene and each target is detected using a target detection method specific to that sensor and target pair. The measured target location is compared against the mapped target location to derive the total extrinsic sensor error:
Extrsensor(R,t)=Σtarget∥RCtarget+t−Dtarget∥2

Where Ctargetis the measured location of the target and Dtargetis the mapped location of the target. We can collect the intrinsic sensor calibration data (as inFIG.10) in a similar way, at each frame of recorded data the targets are detected and intrinsics are collected. These intrinsic sensor calibration data (as inFIG.10) might be the measured distance between pixel coordinates and the lines on a target, or lidar point coordinates and detected planar sides of a target. The total error for a single sensor can be summarized as:
ExtrIntr(R,t,α)sensor=Intrsensor(α)+sensorExtrsensor(R,t)

The weightsensor determines the contribution of that sensor's extrinsic parameter. By collecting the ExtrIntr for every sensor we define a global cost function that describes all intrinsics and extrinsics in the system. We can minimize the total expected error by toggling the calibration parameters for each sensor [R, t, α] via a convex optimization algorithm. The output of the sensor extrinsic calibrations may be a pair of rotation and translation matrices on a per sensor basis with respect to the origin of the 3D space (e.g., as identified via LIDAR).

After the calibration parameters are solved for, tests for the numerical sensitivity of the solution can be performed. This may include, for example, verifying the Jacobian of the solution is near zero in all directions and that the covariance of each parameter is reasonably small (e.g., below a threshold). More sophisticated routines that test for sensitivity to targets and constraints may also be performed.

FIG.12is a flow diagram illustrating operations for interactions between the vehicle and the turntable.

At optional step1205, the turntable405, vehicle102, or surveying system610identifies that the vehicle102is positioned on the platform of the motorized turntable. This may be identified using pressure sensors735of the turntable405, a GNSS or triangulation-based positioning receiver of the vehicle102compared to a known location of the turntable405, image/lidar data captured by the surveying system610indicating that the vehicle102is positioned on motorized turntable405, or some combination thereof. In some cases, the pressure sensors735may be positioned under or beside the guide railing, for example close behind the “stop” wall or incline, to ensure that the vehicle will apply pressure to them. In other cases, the entire turntable is receptive as a pressure sensor. In any case, this information is communicated to the dynamic scene control bridge710and/or the computing device745of the turntable system405, either within the turntable itself (if via sensors735) or via the relevant transceiver(s) ofFIG.7. In some cases, either sensor data capture by the sensors of the vehicle102or rotation of the platform420of the motorized turntable405may automatically begin once the pressure sensors identify that the vehicle102is on the platform420and/or once sensors identify that the vehicle102has stopped moving (e.g., IMU of the vehicle102, regional pressure sensors of regions of the turntable platform420surface, scene surveying system610camera, or some combination thereof). Rotation of the platform420about the base425may occur first before sensor data capture if, for example, calibration is previously designated to start with the vehicle102rotated to a particular rotation orientation or rotation position that is not the same as the rotation orientation or rotation position that the vehicle102is in when it drives (or is otherwise placed) onto the platform420.

In some cases, the rotation of the platform420of the turntable405about the base425via the motors730can manually be triggered instead of being based on, and automatically triggered by, detection of the vehicle at step1205, for example via an input received at the dynamic scene control bridge710and/or the computing device745of the turntable system405from a wired or wireless interface that itself receives an input from a human being, the wired or wireless interface being for example a keyboard or touchscreen or mouse or remote control communicating in a wired or wireless fashion with the dynamic scene control bridge710and/or the computing device745of the turntable system405.

At step1210, one or more motors730of the motorized turntable405are activated to rotate the platform420of the motorized turntable405about the base425of the motorized turntable405(and therefore vehicle102on top of the platform420as well) from a first rotation orientation to a second rotation orientation in response to detection that the vehicle is on the turntable. The one or more motors730may be deactivated, causing the platform of the motorized turntable405(and therefore vehicle102on top of the platform420as well) to stop rotating about the base425with the platform420in the second orientation at the stop of rotation. The term “rotation orientation” may be used to refer to an angle, or angular orientation, or angular position. Other terms may be used in place of the term “rotation position,” such as “angle,” “angular position,” “angular orientation,” “position,” or “orientation.” The first rotation orientation and the second rotation orientation may be a predetermined angle away from each other, for example N degrees, where N is 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, or some amount in between any two of these numbers. The first rotation orientation and the second rotation orientation may be an angle away from each other that is determined by the internal computing system110of the vehicle102, or by the dynamic scene control bridge710, or by the computing device745of the turntable system405, or by some combination thereof, based on which angle would likely be most efficient, comprehensive, or an optimal balance thereof, in completing calibration of the entire fields of view (FOV) of the sensors180of the vehicle102.

At step1215, the vehicle102uses its IMU (or other rotation detection device) to check whether the vehicle102(and therefore the platform420) is still rotating. As the IMU is a collection of accelerometers and/or gyroscopes and/or other motion or rotation detection devices, the vehicle102can alternately separately use accelerometers and/or gyroscopes and/or other motion or rotation detection devices that are among the vehicle102's sensors180to determine this. Alternately, the turntable405may use one or more motion sensors of its own (e.g., accelerometer, gyroscope, IMU, or any other motion sensor discussed herein) to identify whether the platform420of the turntable405is still rotating about the base425. Alternately still, the scene surveying system610may use one or more cameras to visually identify whether the platform of the turntable405and/or the vehicle102is still rotating. In some cases, the device that detects that the vehicle102and/or the platform420of the turntable405has stopped rotating relative to the base425(the vehicle computing system110, the computing device745of the turntable405, the scene surveying system610, and/or the dynamic scene control bridge710) can send a signal identifying the detected stop in rotation to any of the vehicle computing system110, the computing device745of the turntable, the scene surveying system610, or the dynamic scene control bridge710.

If, at step1220, the vehicle102or turntable405or scene surveying system610determines that the rotation has stopped, step1225follows step1220. Otherwise, step1215follows step1220.

In addition, we may use the vehicle102's other sensors180, such as one or more cameras, radar sensors, lidar sensors, sonar sensors, and/or sodar sensors instead of or in addition to the IMU, accelerometers, gyroscopes, and/or motion/rotation detection devices to identify when the vehicle102(and thus the platform420) is still rotating relative to the base425or not. With all of these sensors, rotation may be identified based on whether regions of the calibration environment that should be motionless-walls, the floor, the ceiling, targets that have not been configured and/or commanded to move, light sources620, the scene surveying system610—are changing location between sensor captures (indicating that the vehicle is rotating and/or in motion) or are stationary between sensor captures (indicating that the vehicle is stationary).

At step1225, the vehicle captures sensor data using one or more of its sensors while the vehicle102is at the second position. If, at step1230, the internal computing device110of the vehicle102determines that it has finished capturing sensor data while vehicle is at the second rotational orientation/position, then step1235follows step1230, and optionally, the vehicle computing system110may send a sensor capture confirmation signal to a computing device associated with the turntable405, such as dynamic scene control bridge710and/or the computing device745of the turntable system405. The sensor capture confirmation signal may then be used as a signal that the turntable405is allowed to begin (and/or should begin) rotation of the platform420about the base425from the second rotation orientation to a next rotation orientation. Otherwise, if sensor data capture is not complete step1225follows step1230.

If, at step1235, the internal computing device110of the vehicle102determines that sufficient data has been captured by the vehicle102's sensors180to perform calibration-then no more rotations of the platform420and the vehicle102about the base425are needed and step1240follows step1235, thus proceeding from sensor data capture to sensor calibration. Optionally, the vehicle computing system110may send a sensor capture completion signal to a computing device associated with the turntable405, such as dynamic scene control bridge710and/or the computing device745of the turntable system405. The sensor capture completion signal may then be used as a signal that the platform420of the turntable405is allowed to stop (and/or should stop) rotating about the base425altogether to allow the vehicle102to exit the turntable405and the calibration environment, or that the platform425of the turntable405is allowed to begin (and/or should begin) rotating about the base425to an exit orientation that allow the vehicle102to exit the turntable and the calibration environment (for example when the calibration environment includes many targets around the turntable405except for in an entrance/exit direction, as inFIG.6where an optimal entrance/exit direction is on the bottom-right due to lack of targets and obstacles generally in that direction). Otherwise, if the internal computing device110of the vehicle102does not determine that sufficient data has been captured by the vehicle102's sensors180to perform calibration at step1235, then step1210follows after step1235, to continue rotations of the platform420(and vehicle102) about the base425of the motorized turntable system405. Depending on the sensors180on the vehicle102and the data captured by the sensors180, the sensors180may require one or more full 360 degree rotations of the vehicle102on the platform420, or may require less than one full 360 degree rotation of the vehicle102on the platform420. In one embodiment, sufficient data for calibration of a sensor may mean data corresponding to targets covering at least a subset of the complete field of view of a particular sensor (collectively over a number of captures), with the subset reaching and/or exceeding a threshold percentage (e.g., 30%, 35%, 40%, 45%, 50%, 55%, 60%, 65%, 70%, 75%, 80%, 85%, 90%, 95%, 99%, 100%).

Some sensors may require more data for calibration than others, and thus, one sensor may have captured sufficient data for calibration while another sensor might not. In such cases, step1235may refer to all sensors and thus go through the “NO” arrow if any of the sensors180hasn't captured sufficient data. Alternately, a particular sensor capturing sufficient data, or a majority of sensors capturing sufficient data, may be the deciding factor toward “YES” or “NO.” In some cases, step1235may refer to each sensor separately, and once a particular sensor has captured sufficient data at step1235, that sensor may continue on to step1240for calibration even if the vehicle102on the platform420continues to rotate about the base425and the remaining sensors continue to capture data. Thus, step1235may enable staggered completion of capture of sufficient data for different sensors at different times.

In some cases, sensor data capture and sensor calibration occurs at least partially in parallel; that is, a time period in which sensor data capture occurs may at least partially overlap with a time period in which sensor calibration occurs. In such cases, the sensor may calibrate region by region, for example by calibrating the sensor in one or more regions in which the sensor detects (e.g., “sees”) targets for each data capture until the entire point of view of the sensor, or some sufficient subset is calibrated, with the subset reaching and/or exceeding a threshold percentage (e.g., 30%, 35%, 40%, 45%, 50%, 55%, 60%, 65%, 70%, 75%, 80%, 85%, 90%, 95%, 99%, 100%). Calibration of each sensor may use a different threshold, or certain sensors may share a threshold. When calibration occurs in parallel with capture rather than calibration on the whole sequentially following capture on the whole, step1235can simply identify when calibration of one or more sensors has successfully completed, and treat that as a proxy for identifying whether sufficient data is captured by those sensors to perform calibration.

In this case, however, step1210now rotates the vehicle102from the second position to a third position, and steps1225and1230refer to the third position. The next time step1210is reached in this fashion, it now rotates the vehicle102from the third position to a fourth position, and so on. In this way, step1210rotates the vehicle102on the platform420about the base425from its current rotational orientation/position to its next rotational orientation/position.

At step1240, the internal computing device110of the vehicle102proceeds on from sensor data capture to actual calibration of the sensors, for example as in steps1025-1045ofFIG.10or steps1125-1150ofFIG.11. To clarify, as discussed further above, capturing data via the sensors180as in steps1225-1235and calibrating the sensors180as in step1240can be performed in parallel, between rotations of the platform420and vehicle102about the base425, or in any order that causes the process to be efficient. That is, calibration of data captured by a given one of the sensors180can begin immediately after the given sensor captures any new data, and can continue while the vehicle102is rotating about the base425on the platform420of the turntable405and while the sensors180capture further data. Because calibration and capture may be staggered and/or parallelized so that some further capture occurs after some calibration has started, dashed arrows extend from step1240to steps1210and1225.

It should be understood that many of the steps ofFIG.12(such as1205,1215,1220,1230, and1235) are optional.

FIG.13is a flow diagram illustrating operations for detection of, and compensation for, a sloped turntable surface.

At optional step1305, the turntable405, vehicle102, or surveying system610identifies that the vehicle102is positioned on platform420of the motorized turntable405as in step1205ofFIG.12, for example based on pressure sensors in the turntable, positioning receivers of the vehicle102, and/or on the scene surveying system's visual identification of the vehicle102's position. Step1305(or1205) may optionally include identifying that the vehicle102is specifically in a defined position on the platform420such as the defined position identified inFIG.8B.

At step1310, the one or more sensors180of the vehicle102are used to check whether the vehicle102is level or on a slope. The vehicle102may use an IMU, one or more levels, one or more accelerometers, one or more gyroscopes, one or more lidar sensor, one or more radar sensors, one or more sonar sensors, one or more sodar sensor, one or more cameras, any other sensor discussed herein, or some combination thereof to check whether the vehicle102is level or on a slope. For example, the vehicle102may use one or more gyroscopes, such as one or more gyroscopes found that are part of an IMU of the vehicle102, to compare the angle of the vehicle102while the vehicle102is on the platform420to a reference angle of each gyroscope, the reference angle of the gyroscope corresponding to a level slope. The vehicle102may alternately or additionally use one or more range sensors, such as one or more lidar, radar, sonar, or sodar sensors, to identify the slope of the vehicle102based on on range from each range sensor to different points along the platform420and/or to different points along a floor that the turntable420rests on, where the different points should be equidistant if the vehicle102(and thus the turntable405) is level, or where the different points should have a specific proportional relationship if the vehicle102(and thus the turntable405) is level. The vehicle102may alternately or additionally use images from one or more cameras, for example to identify a horizon in the one or more images, to identify a slope of the vehicle102(and thus the turntable405). With cameras, a slope of the floor may be detectable by checking slopes of various edges of the calibration environment that should be parallel or perpendicular in X, Y, or Z dimensions (e.g., edge between floor and wall, edge between one wall and another wall, edge between wall and ceiling) against each other to see if any unexpected angles emerge, which may indicate a slope. Sensors735within the turntable405itself, such as any of those described above with respect to the vehicle102in step1310, may alternately or additionally be used to detect the slope of the turntable405(and thus the vehicle) instead of or alongside the sensors180of the vehicle102.

If, at step1315, the internal computing device110of the vehicle102identifies that the vehicle102(and therefore the turntable405) is level while vehicle is at this position, then step1315is followed by step1325; otherwise, step1315is followed by step1320. If the vehicle102(and/or turntable405) uses a gyroscope measurement to determine slope, then the gyroscope angle measurement can be compared to a gyroscope reference angle of the gyroscope, and if a difference between the two is found to exceed an angle threshold (e.g., N degrees, where N is 0, 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6, 6.5, 7, 7.5, 8, 8.5, 9, 9.5, or 10) then the slope is not level. If the vehicle102(and/or turntable405) uses a range sensor to determine slope, then one or more differences from expected range values to different points along the floor exceeding a range threshold (e.g, N millimeters or centimeters, where N is 0, 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6, 6.5, 7, 7.5, 8, 8.5, 9, 9.5, or 10) may indicate that the slope is not level. If the vehicle102(and/or turntable405) uses a camera to determine slope, then differences from expected angles in resulting images that are found to exceed an angle threshold (e.g., N degrees, where N is 0, 0.5, 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6, 6.5, 7, 7.5, 8, 8.5, 9, 9.5, or 10) may indicate that the slope is not level. At step1320, the internal computing device110of the vehicle102(and/or the computing device745of the motorized turntable system405) identifies a specific slope of vehicle102(and therefore a slope of the turntable405) at the vehicle and turntable's current rotation position. At step1325, the vehicle captures sensor datasets using its sensors180, optionally at multiple orientations, for example as in steps910,1010,1110,1115, and/or1225. In some cases, the slope may have already been identified at step1310, at which case step1325may not entail performing anything. In other cases, the slope may have been identified in a quick, imprecise manner at step1310, while at step1325, a more accurate measurement is determined, using any of the sensors of the vehicle102and/or of the motorized turntable system405discussed with respect to step1310. In some cases, for example, the motorized turntable405may rotate the platform420about the base425while the vehicle102is on the platform420, and the vehicle102may perform additional measurements at different orientations along this rotation and use those additional measurements to more precisely determine slope. These additional slope detection measurements at different orientations may occur at step1310and/or at step1325, and may occur while the platform420is stationary relative to the base425(for example during pauses in rotation), while the platform420is rotating relative to the base425, or some combination thereof.

At step1330, the vehicle102performs sensor calibration of its sensors, for example as in steps915-925,1025-1040,1125-1150, and/or1240. The sensor calibration performed at step1330may be performed partially (e.g., more sensor calibration capture data is yet to be collected, and/or the motorized turntable still has more rotations to perform) or to completion.

Note that, as discussed with respect toFIG.12, capture of data by the sensors180of the vehicle102may occur in parallel with calibration of the sensors180of the vehicle102. This may cause calibration and capture to be staggered and/or parallelized so that some further capture occurs after some calibration has started, represented by the dashed arrow from step1330to step1325.

At step1335, the internal computing device110of the vehicle102additionally compensates for the slope identified at step1320in its calibration of sensors at step1330. For example, if the turntable is not level surface, but is instead tilted along a north-south axis by one degree, then all of the sensor results will be skewed along the north-south axis. The internal computing device110of the vehicle102can calculate new positions of everything in the calibration environment by tilting everything by one degree along the north-south axis in the opposite direction as the slope identified in step1320.

At optional step1340, the rotation from the current position to the next position may be triggered, much like in step1210ofFIG.12. Step1340may be triggered when the calibration performed at step1330is only partial rather than complete, or when the slope is still not determined to a satisfactory degree (i.e., other angles should be checked). The rotation of step1340may be followed by step1310, detecting slope again after the rotation, which may be helpful since certain slopes may be more detectable while the vehicle102is in one orientation versus another orientation due to the fact that vehicles102are typically longer along their forward-backward axis than they are wide along their left-right axis and/or simply due to limitations of the sensor(s) used to determine slope at certain orientations that might not be limiting at others. Detection of the slope may be detected, verified, and/or adjusted by the vehicle102and/or by the turntable system405while the platform420is at various orientations with respect to the base425. Thus, slope detection may occur in parallel with other calibration procedures illustrated in and/or discussed with respect toFIG.9,FIG.10,FIG.11,FIG.12, and/orFIG.14. By repeating this slope detection and compensation at a multiple rotation locations, the internal computing device110of the vehicle102may develop a more nuanced understanding of how level or sloped the vehicle102, and thus the turntable405, is at each position.

FIG.14Ais a flow diagram illustrating operations for interactions between the vehicle and a lighting system.

At step1405, the vehicle102captures sensor datasets using its sensors180, for example as in steps910,1010,1110,1115,1225, and/or1325. At step1410, the internal computing system110of the vehicle102identifies whether a characteristic of one or more sensor targets—in this case lighting conditions in at least one area of the calibration environment that includes one or more sensor targets—are suboptimal, at least for the purposes of calibration. In some cases, the computer110of the vehicle102may identify that a representation of a sensor target that is identified within a sensor dataset (such as a photo or video) captured using the sensor (such as a camera) is suboptimal or not suitable for calibration, for example because the sensor target is too dimly lit, too brightly lit, or lit from the wrong angle (e.g., causing glare, shadows, dimness, brightness, uneven lighting, or otherwise affecting the representation of the sensor target). Such lighting conditions may be suboptimal because they may cause a sensor to not properly or clearly detect out one or more features of the sensor target, such as a checkerboard pattern210A or ArUco pattern210B or crosshair pattern210C of a camera target200, or a shape215of a radar target220, or a aperture225/240and/or marking230and/or target ID235of a combined camera/depth sensor target250.

If, at step1410, the computer110of the vehicle102determines that the lighting conditions are suboptimal, then step1410is followed by step1415; otherwise, step1410is followed by step1420, at which point the vehicle proceeds from capture to sensor calibration of its sensors, for example as in steps915-925,1025-1040,1125-1150,1240, and/or1330.

Note that, as discussed with respect toFIG.12, capture of data by the sensors180of the vehicle102may occur in parallel with calibration of the sensors180of the vehicle102. This may cause calibration and capture to be staggered and/or parallelized so that some further capture occurs after some calibration has started, represented by the dashed arrow from step1420to step1405.

At step1415, the internal computing system110of the vehicle102sends an environment adjustment signal or message to an environment adjustment system (in this case the lighting system760) to activate one or more actuators762and thereby adjust lighting conditions in the at least one area (and affecting one or more sensor targets in the at least one area) of the calibration environment. The one or more actuators762may control one or more motors associated with the lighting system760, one or more switches associated with the lighting system760, and/or one or more dimmers associated with the lighting system760. Upon receiving the environment adjustment signal or message from the vehicle102, the lighting system760can activate the one or more actuators762, and can thereby effect a modification to the characteristic (i.e., the lighting condition) of the one or more sensor targets, for example by brightening one or more light sources620, by dimming one or more light sources620, by moving one or more light sources620translationally, by rotating one or more light sources620(i.e., moving the one or more light sources620rotationally), by activating (i.e., turning on) one or more light sources620, by deactivating (i.e., turning off) one or more light sources620, by changing a color emitted by (or filtered via color filters applied to) the one or more light sources620, by otherwise modifying the one or more light sources620, or some combination thereof. Note that an increase in brightness as discussed herein may refer to brightening one or more light sources620, activating one or more one or more light sources620, and/or moving one or more light sources620. Note that a decrease in brightness as discussed herein may refer to dimming one or more light sources620, deactivating one or more one or more light sources620, and/or moving one or more light sources620.

After step1415, the process returns to1405to capture the sensor data with newly-adjusted (i.e., optimized) lighting. The newly-adjusted lighting is then checked at step1410to see whether the adjustment from step1415corrected the lighting condition issue identified previously at step1410(leading to step1420), or if further adjustments are required (leading to step1415once again).

FIG.14Bis a flow diagram illustrating operations for interactions between the vehicle and a target control system.

At step1425, the vehicle102captures sensor datasets using its sensors180, for example as in steps910,1010,1110,1115,1225,1325, and/or1405. At step1430, the internal computing system110of the vehicle102identifies whether a characteristic of one or more sensor targets—in this case sensor target positioning of at least one target in the calibration environment is suboptimal, at least for the purposes of calibration. In some cases, the computer110of the vehicle102may identify that a representation of a sensor target that is identified within a sensor dataset (such as a photo or video or radar image/video or lidar image/video) captured using the sensor (such as a camera or radar or lidar sensor) is suboptimal or not suitable for calibration, for example because the sensor target is located in a position and/or facing an orientation in which the sensor cannot properly or clearly detect out one or more features of the sensor target, such as a checkerboard pattern210A or ArUco pattern210B or crosshair pattern210C of a camera target200, or a shape215of a radar target220, or a aperture225/240and/or marking230and/or target ID235of a combined camera/depth sensor target250.

If, at step1430, the computer110of the vehicle102determines that the sensor target positioning is sub-optimal, then step1430is followed by step1435; otherwise, step1430is followed by step1440, at which point the vehicle proceeds from capture to sensor calibration of its sensors, for example as in steps915-925,1025-1040,1125-1150,1240,1330, and/or1420.

Note that, as discussed with respect toFIG.12, capture of data by the sensors180of the vehicle102may occur in parallel with calibration of the sensors180of the vehicle102. This may cause calibration and capture to be staggered and/or parallelized so that some further capture occurs after some calibration has started, represented by the dashed arrow from step1440to step1425.

At step1435, the internal computing system110of the vehicle102sends an environment adjustment signal or message to an environment adjustment system (in this case the target control system770) to activate one or more actuators774and thereby move the at least one sensor target to a more optimal position in the calibration environment. The one or more actuators774may control one or more motors associated with the target control system770and/or one or more switches associated with the target control system770. Upon receiving the environment adjustment signal or message from the vehicle102, the target control system770can activate the one or more actuators774, and can thereby effect a modification to the characteristic (i.e., the positioning) of the one or more sensor targets, for example by activating one or more motors that translationally move one or more targets and/or by activating one or more motors that rotate one or more targets (each about an axis).

After step1435, the process returns to1425to capture the sensor data with newly-moved (i.e., optimized) sensor target positioning. The newly-moved target positioning is then checked at step1430to see whether the adjustment from step1435corrected the target positioning issue identified previously at step1430(leading to step1440), or if further adjustments are required (leading to step1435once again).

In some cases, the adjustment(s) to lighting ofFIG.14Aand the adjustment(s) to target positioning ofFIG.14Bmay both occur following capture of the same sensor dataset with the same sensor, In such cases, the checks of steps1410and1430may be performed repeatedly, once after each adjustment in either target positioning or lighting, since movement of a sensor target may correct or ameliorate issues with lighting, and on the other hand, adjustment of lighting may also correct or ameliorate issues with target positioning. In such cases, the sending of messages, and the resulting adjustments, of steps1415and step1435, can either be performed sequentially (and then tested at steps1410and/or1430), or can be performed in parallel (and then tested at steps1410and/or1430).

While various flow diagrams provided and described above, such as those inFIGS.9,10,11,12,13,14A, and14B, may show a particular order of operations performed by some embodiments of the subject technology, it should be understood that such order is exemplary. Alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, or some combination thereof. It should be understood that unless disclosed otherwise, any process illustrated in any flow diagram herein or otherwise illustrated or described herein may be performed by a machine, mechanism, and/or computing system #00discussed herein, and may be performed automatically (e.g., in response to one or more triggers/conditions described herein), autonomously, semi-autonomously (e.g., based on received instructions), or a combination thereof. Furthermore, any action described herein as occurring in response to one or more particular triggers/conditions should be understood to optionally occur automatically response to the one or more particular triggers/conditions.

As described herein, one aspect of the present technology is the gathering and use of data available from various sources to improve quality and experience. The present disclosure contemplates that in some instances, this gathered data may include personal information. The present disclosure contemplates that the entities involved with such personal information respect and value privacy policies and practices, for example by encrypting such information.

FIG.15shows an example of computing system1500, which can be for example any computing device making up internal computing system110, remote computing system150, (potential) passenger device executing rideshare app170, or any component thereof in which the components of the system are in communication with each other using connection1505. Connection1505can be a physical connection via a bus, or a direct connection into processor1510, such as in a chipset architecture. Connection1505can also be a virtual connection, networked connection, or logical connection.

In some embodiments, computing system1500is a distributed system in which the functions described in this disclosure can be distributed within a datacenter, multiple data centers, a peer network, etc. In some embodiments, one or more of the described system components represents many such components each performing some or all of the function for which the component is described. In some embodiments, the components can be physical or virtual devices.

Example system1500includes at least one processing unit (CPU or processor)1510and connection1505that couples various system components including system memory1515, such as read-only memory (ROM)1520and random access memory (RAM)1525to processor1510. Computing system1500can include a cache of high-speed memory1512connected directly with, in close proximity to, or integrated as part of processor1510.

Processor1510can include any general purpose processor and a hardware service or software service, such as services1532,1534, and1536stored in storage device1530, configured to control processor1510as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor1510may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction, computing system1500includes an input device1545, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech, etc. Computing system1500can also include output device1535, which can be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input/output to communicate with computing system1500. Computing system1500can include communications interface1540, which can generally govern and manage the user input and system output. The communication interface may perform or facilitate receipt and/or transmission wired or wireless communications via wired and/or wireless transceivers, including those making use of an audio jack/plug, a microphone jack/plug, a universal serial bus (USB) port/plug, an Apple® Lightning® port/plug, an Ethernet port/plug, a fiber optic port/plug, a proprietary wired port/plug, a BLUETOOTH® wireless signal transfer, a BLUETOOTH® low energy (BLE) wireless signal transfer, an IBEACON® wireless signal transfer, a radio-frequency identification (RFID) wireless signal transfer, near-field communications (NFC) wireless signal transfer, dedicated short range communication (DSRC) wireless signal transfer, 802.11 Wi-Fi wireless signal transfer, wireless local area network (WLAN) signal transfer, Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), Infrared (IR) communication wireless signal transfer, Public Switched Telephone Network (PSTN) signal transfer, Integrated Services Digital Network (ISDN) signal transfer, 3G/4G/5G/LTE cellular data network wireless signal transfer, ad-hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or some combination thereof. The communications interface1540may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers that are used to determine a location of the computing system1500based on receipt of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US-based Global Positioning System (GPS), the Russia-based Global Navigation Satellite System (GLONASS), the China-based BeiDou Navigation Satellite System (BDS), and the Europe-based Galileo GNSS. There is no restriction on operating on any particular hardware arrangement, and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

Storage device1530can be a non-volatile and/or non-transitory and/or computer-readable memory device and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, a floppy disk, a flexible disk, a hard disk, magnetic tape, a magnetic strip/stripe, any other magnetic storage medium, flash memory, memristor memory, any other solid-state memory, a compact disc read only memory (CD-ROM) optical disc, a rewritable compact disc (CD) optical disc, digital video disk (DVD) optical disc, a blu-ray disc (BDD) optical disc, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Stick® card, a smartcard chip, a EMV chip, a subscriber identity module (SIM) card, a mini/micro/nano/pico SIM card, another integrated circuit (IC) chip/card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cache memory (L1/L2/L3/L4/L5/L #), resistive random-access memory (RRAM/ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and/or a combination thereof.

The storage device1530can include software services, servers, services, etc., that when the code that defines such software is executed by the processor1510, it causes the system to perform a function. In some embodiments, a hardware service that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor1510, connection1505, output device1535, etc., to carry out the function.

For clarity of explanation, in some instances, the present technology may be presented as including individual functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software.

Any of the steps, operations, functions, or processes described herein may be performed or implemented by a combination of hardware and software services or services, alone or in combination with other devices. In some embodiments, a service can be software that resides in memory of a client device and/or one or more servers of a content management system and perform one or more functions when a processor executes the software associated with the service. In some embodiments, a service is a program or a collection of programs that carry out a specific function. In some embodiments, a service can be considered a server. The memory can be a non-transitory computer-readable medium.

In some embodiments, the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.

Methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer-readable media. Such instructions can comprise, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The executable computer instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and/or information created during methods according to described examples include magnetic or optical disks, solid-state memory devices, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.

Devices implementing methods according to these disclosures can comprise hardware, firmware and/or software, and can take any of a variety of form factors. Typical examples of such form factors include servers, laptops, smartphones, small form factor personal computers, personal digital assistants, and so on. The functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.

The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.

Although a variety of examples and other information was used to explain aspects within the scope of the appended claims, no limitation of the claims should be implied based on particular features or arrangements in such examples, as one of ordinary skill would be able to use these examples to derive a wide variety of implementations. Further and although some subject matter may have been described in language specific to examples of structural features and/or method steps, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or acts. For example, such functionality can be distributed differently or performed in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.