Patent Description:
Accurate and consistent obstacle detection and navigation can be key elements of autonomous driving applications. Typically, an autonomous vehicle utilizes various on-board sensors to detect obstacles, other aspects of the roadway, and/or other aspects of an environment around the vehicle, which can be referred to as "perception information" or "perception data" representing what an ordinary driver would perceive in the surrounding environment of a vehicle. Examples of such sensors include, for example, one or more of vision sensors (e.g., camera(s)), radio detection and ranging (i.e., radar) sensors, and/or light detection and ranging (i.e., LiDAR) sensors.

The perception information detected by the on-board sensors is processed and analyzed by image analysis software or a perception system to identify the objects surrounding the vehicle. The objects may include, for example, traffic signals, roadway boundaries, other vehicles, pedestrians, and/or obstacles. The perception system may also determine, for one or more identified objects in an environment, the current state of the object. The state information may include, for example, an object's current speed and/or acceleration, current orientation, size/footprint, type (e.g., vehicle vs. pedestrian vs. bicycle vs. static object or obstacle), and/or other state information. Perception systems known in the art apply different combinations of object recognition algorithms, video tracking algorithms, and computer vision algorithms (e.g., track objects frame-to-frame iteratively over a number of time periods) to determine the information about objects and/or to predict future location of objects from captured perception information and/or sensor data. In order to fully test or evaluate image analysis software and perception systems for vehicles, such as autonomous vehicles, for different cases or scenarios, there is a need to generate image datasets including obstruction cases that emulate real-world conditions. Different analysis software and systems can be tested and evaluated based on their ability to correctly detect or identify such obstruction cases. The devices, methods, and systems of the present disclosure are provided to assist in generating such image datasets for use in testing and evaluating image analysis software and perception systems. <CIT> describes an imaging device including an image sensor, a control unit coupled to the image sensor, and a light blocking element coupled to the control unit. The control unit is configured to adjust the light blocking element in response to image information received from the image sensor. The light blocking element is further configured to regulate light received at the image sensor. <CIT> describes an adaptive filter system and a method for controlling the adaptive filter system. The system includes one or more filters to attenuate incoming light and the one or more filters can be moved by one or more actuators. <CIT> describes sensors coupled to a vehicle being calibrated using a dynamic scene with sensor targets around a motorized turntable that rotates the vehicle to different orientations. The sensors capture data at each vehicle orientation along the rotation and a vehicle's computer identifies representations of the sensor targets within the data captured by the sensors, and calibrates the sensor based on these representations.

Advantageous embodiments are described in the dependent claims, the following description and the drawings. According to an aspect of the present disclosure, a mounting device for selectively positioning an optical element with an obstruction within a field-of-view of an optical sensor of a vehicle includes:
a housing defining an opening sized to fit over an aperture of the optical sensor; a holder for the optical element connected to the housing and positioned such that, when the holder is in a first position, the optical element is at least partially within the field-of-view of the optical sensor; and a motorized actuator. The motorized actuator can be configured to move the holder to adjust the position of the optical element relative to the field-of-view of the optical sensor.

According to another aspect of the present disclosure, a kit of parts includes at least one of the previously described motorized mounting devices and a plurality of optical elements. Each of the optical elements are configured to be removably mounted to the holder of one of the motorized mounting devices.

According to another aspect of the present disclosure, a vehicle, such as a fully autonomous vehicle or a semi-autonomous vehicle, includes a plurality of optical sensors positioned to obtain images of objects and/or an environment surrounding the vehicle, and a plurality of the previously described motorized mounting devices. Each mounting device is positioned over the aperture of one of the plurality of optical sensors and is configured to move the optical element attached to the holder of the mounting device relative to the field-of-view of the respective optical sensor.

According to another aspect of the present disclosure, a method for testing or evaluating image analysis software or a perception system of a vehicle that analyzes images collected by at least one optical sensor of the vehicle is provided. The method includes: obtaining image data of objects and/or an environment surrounding the vehicle from the at least one optical sensor of the vehicle; as the image data is being captured, causing a motorized mounting device, such as any of the previously described motorized mounting devices, to move the holder to position the optical element at least partially within the field-of-view of the at least one optical sensor, such that at least a portion of the obtained image data is distorted by the optical element; and analyzing the obtained image data using the image analysis software or perception system for analysis of images collected by the at least one optical sensor of a vehicle to determine whether the image analysis software or perception system correctly identifies portions of the image data that are distorted by the optical element.

Additional advantages and details are explained in greater detail below with reference to the exemplary embodiments that are illustrated in the accompanying schematic figures, in which:.

The following description is provided to enable those skilled in the art to make and use the described embodiments contemplated for carrying out the invention. Various modifications, equivalents, variations, and alternatives, however, will remain readily apparent to those skilled in the art. Any and all such modifications, variations, equivalents, and alternatives are intended to fall within the spirit and scope of the present invention.

However, it is to be understood that the invention may assume alternative variations and step sequences, except where expressly specified to the contrary. It is also to be understood that the specific devices and processes illustrated in the attached drawings, and described in the following specification, are simply exemplary embodiments of the invention.

No aspect, component, element, structure, act, step, function, instruction, and/or the like used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles "a" and "an" are intended to include one or more items, and may be used interchangeably with "one or more" and "at least one. " As used herein, the terms "has," "have," "having," or the like are intended to be open-ended terms. Further, the phrase "based on" is intended to mean "based at least partially on" unless explicitly stated otherwise.

As used herein, the term "communication" may refer to the reception, receipt, transmission, transfer, provision, and/or the like, of data (e.g., information, signals, messages, instructions, commands, and/or the like). For one unit (e.g., a device, a system, a component of a device or system, combinations thereof, and/or the like) to be in communication with another unit means that the one unit is able to directly or indirectly receive information from and/or transmit information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, and/or the like) that is wired and/or wireless in nature. Additionally, two units may be in communication with each other even though the information transmitted may be modified, processed, relayed, and/or routed between the first and second unit. For example, a first unit may be in communication with a second unit even though the first unit passively receives information and does not actively transmit information to the second unit. As another example, a first unit may be in communication with a second unit if at least one intermediary unit processes information received from the first unit and communicates the processed information to the second unit.

With reference to the figures, the present disclosure is directed, in part, to a mounting device <NUM> for selectively positioning an optical element <NUM> within a field-of-view of an optical sensor <NUM>. The mounting device <NUM> can be used with one or more vision sensors or cameras of a vehicle <NUM>, such as a fully autonomous or semi-autonomous vehicle, to create datasets of captured images including distortions provided by the optical element <NUM>. In some examples, the vehicle <NUM> can include multiple optical sensors <NUM>, such as an array of multiple cameras, with different mounting devices <NUM> positioned over apertures and/or within a field-of-view of each of the multiple optical sensors <NUM>.

The mounting device <NUM> selectively positions the optical element <NUM> within the field-of-view of the optical sensor <NUM> by, for example, pivoting, sliding, swinging, or otherwise moving the optical element <NUM> towards, or away from, the optical sensor <NUM> as image data is being collected by the optical sensor <NUM>. The optical sensors <NUM> or cameras can be positioned at any convenient location on the vehicle <NUM>, such as, for example, on a sensor frame, housing, tripod, or another support structure that positions optical sensors <NUM> or cameras to obtain different views of an environment surrounding the vehicle <NUM>. For example, optical sensors <NUM> or cameras can be positioned to obtain a <NUM> degree, <NUM> degree, or <NUM> degree panoramic views of the environment. As described in further detail herein, the optical element <NUM> can be a filter or lens including an obstruction or another feature that distorts images captured by the optical sensor <NUM>. By selectively positioning the optical element <NUM> within the field-of-view of the optical sensors <NUM>, image data can be generated that includes the distortions of the filter or lens created on demand, such that the captured dataset includes distorted images at known or predetermined times and for predetermined durations. The captured dataset can be processed and analyzed to test image analysis software and/or a vehicle perception system and, in particular, to determine whether the software or perception system correctly detects and/or appropriately classifies image data captured when the optical element <NUM> is in place within the field-of-view of the optical sensor <NUM>.

In the device applied for, the optical element <NUM> can be selected to emulate or represent obstructions present in image data collected during real-world use of the vehicle <NUM> and associated cameras. For example, as described in further detail herein, the image analysis software or system can be configured to detect when obstructions (i.e., dust, dirt, waste, leaves, mud, scratches, bird poop, insects, rain drops, etc.) are present in captured image data. Once the software or perception system determines that certain distortions in captured images are caused by obstructions and not, for example, by an oncoming obstacle, appropriate action can be taken. In other examples, the optical elements <NUM> can include light filters to approximate driving scenarios that cause significant and/or sudden changes in light intensity or brightness in captured images as occurs, for example, when the vehicle exits a tunnel. By applying different image analysis algorithms to the captured datasets, users can evaluate which software or perception systems correctly identify or trigger for portions of the dataset collected when the optical element <NUM> is positioned within the field-of-view of the sensor <NUM>. Accordingly, the motorized mounting devices <NUM> of the present disclosure can be used to generate useful datasets of captured images. The datasets can be processed or analyzed to test or optimize image analysis software and perception systems.

With reference to <FIG>, a vehicle <NUM>, such as a fully autonomous vehicle or a semi-autonomous vehicle, is illustrated including a sensor housing or sensor frame <NUM> for supporting an array of optical sensors <NUM> and other object-detection sensors for detecting objects and/or the environment surrounding the vehicle <NUM>. As shown in <FIG>, the vehicle <NUM> is an automobile, such as a four-door sedan. In other examples, within the scope of the present disclosure, the vehicle <NUM> can be any other moving form of conveyance that is capable of carrying either human occupants and/or cargo and is powered by any form of energy. For example, the "vehicle" can be a car, truck, van, train, fully autonomous vehicle, semi-autonomous vehicle, bicycle, moped, motorcycle, aircraft, aerial drone, water-going vessel, ship, or boat. As used herein, an "autonomous vehicle" refers to a vehicle having a processor, programming instructions, and drivetrain components that are controllable by the processor without requiring a human operator. An autonomous vehicle may be "fully autonomous" in that it does not require a human operator for most or all driving conditions and functions, or it may be "semi-autonomous" in that a human operator may be required in certain conditions or for certain operations, or that a human operator may override the vehicle's autonomous system and may take control of the vehicle <NUM>.

The vehicle <NUM> includes a vehicle body <NUM> and windows <NUM> enclosing a cabin <NUM>. As will be appreciated by those skilled in the art, the vehicle <NUM> can include multiple mechanical, electrical, and/or monitoring systems for moving, controlling, and monitoring a condition of the vehicle <NUM> while in use. For example, as shown schematically in <FIG>, the vehicle <NUM> can include an engine or motor <NUM> and various sensors for measuring parameters of the vehicle <NUM>. In gas-powered or hybrid vehicles having a fuel-powered engine <NUM>, the sensors may include, for example, an engine temperature sensor <NUM>, a battery voltage sensor <NUM>, an engine rotations-per-minute ("RPM") sensor <NUM>, a throttle position sensor <NUM>, and other sensors as are known in the autonomous vehicle art. In an electric or hybrid vehicle, the vehicle <NUM> may have an electric motor and sensors, such as battery monitoring sensor (e.g., to measure current, voltage, and/or temperature of the battery), motor current sensor, motor voltage sensor, and/or motor position sensors, such as resolvers and encoders. The vehicle <NUM> can also include location sensors <NUM> (e.g., a Global Positioning System ("GPS") device) for tracking vehicle location. The vehicle <NUM> can also include an on-board perception system <NUM> for analyzing data from the sensors, in particular, data from the optical sensors <NUM> to detect objects and obstacles in proximity to the vehicle <NUM>.

As shown in <FIG> and <FIG>, the vehicle <NUM> further comprises the sensor housing or frame <NUM> for supporting the array of optical sensors <NUM> including openings or apertures <NUM> positioned around the frame <NUM>. As shown in <FIG>, the housing or frame <NUM> is positioned on the roof of the vehicle <NUM>. In other examples, the frame <NUM> and/or other structures for supporting optical sensors and other vision sensors can be positioned at many other locations either on the exterior of the vehicle body <NUM> or inside the vehicle cabin <NUM>. For example, various camera supporting structures, such as tripods, scaffolds, frames, clamps, brackets, housings, and similar support structures, can be used with the vehicle <NUM> and motorized mounting devices <NUM> of the present disclosure. An exemplary frame <NUM> for supporting cameras and other optical and/or vision sensors on a vehicle <NUM>, referred to as the Tiara, which can be used with the motorized mounting devices <NUM> of the present disclosure, is manufactured by Argo AI of Pittsburgh, Pennsylvania. Those skilled in the art will appreciate that the mounting devices <NUM> of the present disclosure can also be adapted for use with many other optical sensors, cameras, and support structures within the scope of the present disclosure.

The optical sensors <NUM> contained within and/or supported by the sensor housing or frame <NUM> can include components of a conventional digital camera, RGB camera, digital video camera, red-green-blue sensor, far-field camera, and/or depth sensor for capturing visual information and static or video images, as are known in the art. The components of the optical sensors <NUM> or camera can be adapted to fit within and/or to be mounted to the sensor housing or frame <NUM>. As is known in the art, a digital camera generally includes a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) imaging sensor, lens(es), a multifunctional video control chip, and a set of discrete components (e.g., capacitor, resistors, and connectors). An image or series of images is recorded by the imaging sensor and can be processed by the video control chip. Captured images can be stored on computer memory associated with the camera or with other data collection systems of the vehicle <NUM>, such as the data storage device <NUM> (shown in <FIG>). In some examples, the camera or optical sensors <NUM> can also include multiple image capture features for obtaining stereo images of the environment around the vehicle <NUM>. The stereo-image can be processed to determine depth information for objects in the environment surrounding the vehicle <NUM>. In other examples, the camera can be a wide angle or fish-eye camera, a three-dimensional camera, a light-field camera, or similar devices for obtaining images, as are known in the art.

As shown in <FIG>, the sensor housing or frame <NUM> includes the multiple openings or apertures <NUM> for the optical sensors <NUM> or cameras positioned around the sensor housing or frame <NUM>. The optical sensors <NUM> and apertures <NUM> can be oriented in different directions to provide a panoramic view (i.e., a view of from <NUM> degrees to <NUM> degrees) of objects and/or the environment surrounding the vehicle <NUM>. The motorized mounting devices <NUM> holding the optical elements <NUM> are positioned proximate to the openings or apertures <NUM> for moving the optical elements <NUM> over or away from the apertures <NUM>. As described in further detail herein, the motorized mounting devices <NUM> can include a device housing <NUM> that is mounted to an outer surface of the frame <NUM> by, for example, double-sided tape, adhesives, fasteners, or other mechanical or chemical connectors, as are known in the art. In <FIG> and <FIG>, there is a motorized mounting device <NUM> proximate to each aperture <NUM>. In other examples, some of the apertures <NUM> can include mounting devices <NUM>, while others do not, so that image data can be captured both with and without visual distortion.

In some examples, the multiple motorized mounting devices <NUM> can be in electrical communication and/or connected together so that they can be operated simultaneously and/or by the same controller. For example, each mounting device <NUM> can be in wired connection with a receiver box or receiving device <NUM> mounted to the sensor housing or frame <NUM>. As described in further detail herein, the receiver box or receiving device <NUM> can be in wired or wireless connection with a remote control <NUM> (shown in <FIG> and <FIG>), allowing a user to actuate the motorized mounting devices <NUM> independently, simultaneously, sequentially, or in any other manner.

The vehicle <NUM> can also include a LiDAR sensor system including, for example, cylindrical LiDAR detectors <NUM> mounted to a top portion of the housing or frame <NUM>. In some examples, the frame <NUM> can also contain sensors for a radar and/or sonar system (not shown). The frame <NUM> can also contain environmental sensors (not shown), such as a precipitation sensor and/or ambient temperature sensor.

As previously described, the motorized mounting devices <NUM> are mounted proximate to the apertures <NUM> and optical sensors <NUM>, and are configured for selectively positioning optical elements <NUM> within the field-of-view of an optical sensors <NUM>. Examples of the motorized mounting devices <NUM> are shown in <FIG>. As shown in <FIG> and <FIG>, the motorized mounting device <NUM> can include three separate components, namely a holder <NUM> for holding the optical element <NUM> in position over the optical sensor <NUM>, the housing <NUM> defining an opening, central bore, or interior <NUM> sized to fit over the aperture <NUM> of the optical sensor <NUM>, and a motorized actuator <NUM> for moving the holder <NUM> between, for example, a first (i.e., a partially closed or fully closed) position, as shown in <FIG>, and a second (i.e., a fully open or partially open) position, as shown in <FIG>. As used herein, the "first position" or the "closed position" can be a position in which the optical element <NUM> is at least partially within the field-of-view and/or at least partially blocks the field-of-view of the optical sensor <NUM>. The "second position" or the "open position" can be a position where a larger area of the optical element <NUM> is outside of the field-of-view of the optical sensor <NUM> than in the first or closed position and/or where a smaller portion of the field-of-view of the optical sensor <NUM> is blocked by the optical element <NUM>, than when the optical element <NUM> is in the first or closed position.

The housing <NUM> can be a generally annular structure formed from a rigid material, such as hard plastic or metal, sized to fit over the aperture <NUM> or opening of the optical sensor <NUM>. As shown in <FIG>, the housing <NUM> can include a first end <NUM> configured to be positioned over or proximate to the aperture <NUM> of the optical sensor <NUM>, a second end <NUM> opposite the first end <NUM>, and a sidewall <NUM> extending between the first end <NUM> and the second end <NUM>. In some examples, the sidewall <NUM> includes or defines a tapered annular inner surface <NUM> extending from the first end <NUM> to the second end <NUM> of the housing <NUM>.

The holder <NUM> is an annular structure configured to receive and securely support the optical element <NUM>. The holder <NUM> can be formed from the same rigid material, such as metal or hard plastic, as the housing <NUM>. Dimensions of the housing <NUM> and holder <NUM> can be selected to correspond to specific dimensions of the optical sensor <NUM> and aperture <NUM>. For example, an inner diameter ID1 (shown in <FIG>) of the first end <NUM> of the housing <NUM> can be selected to correspond to or match a diameter of the aperture <NUM> of the optical sensor <NUM>. In a similar manner, the inner diameter ID2 (shown in <FIG>) of the second end <NUM> of the housing and/or an inner diameter ID3 (shown in <FIG>) of the holder <NUM> can be selected to correspond to an outer diameter of the optical element <NUM>. In some examples, the housing <NUM> and/or holder <NUM> can be made by additive manufacturing or three-dimensional (3D) printing to a custom size selected based on specific requirements of the sensor housing or frame <NUM>, optical sensor <NUM>, and optical element <NUM>. In other examples, as described in further detail herein, adapter plate(s) <NUM> can be provided so that the housing <NUM> can be used with optical sensors <NUM> and apertures <NUM> of different sizes.

As shown in <FIG>, the holder <NUM> can be hingedly connected to the housing <NUM> at a hinge <NUM>, which allows the holder <NUM> to swing towards or away from the housing <NUM>, as shown by arrow A1 (in <FIG> and <FIG>). The holder <NUM> is positioned such that, when in the first or closed position (shown in <FIG>), the optical element is at least partially within the field-of-view of the optical sensor <NUM>. As used herein, "at least partially within the field-of-view" means that, when the holder <NUM> is in the first or closed position, at least a portion of an image captured by the optical sensor <NUM> is captured through the optical element <NUM> and, desirably, includes visual distortions of the optical element <NUM>. In some examples, the holder <NUM> positions the optical element <NUM> to fully cover the optical sensor <NUM> meaning that the entire image captured by the optical sensor <NUM> is captured through the optical element <NUM>. In other examples, portions (e.g., <NUM>%, <NUM>%, <NUM>%, or <NUM>%) of the captured image is through the optical element <NUM>, while other portions of the captured image are free from distortions of the optical element <NUM>.

The optical element <NUM> is generally a lens or filter, such as a lens or filter that is used with a conventional digital camera, adapted to include the visual obstructions or distortions described herein. For example, the optical element <NUM> can include a lens <NUM> (shown in <FIG>), such as an <NUM> camera lens or a lens of any other convenient size. The optical element <NUM> can be sized to be press-fit into the holder <NUM>, thereby securely fixing the optical element <NUM> to the mounting device <NUM>. In order to change optical elements <NUM>, in some examples, a user can use a small flat tool (such as a small precision flat head screwdriver) to gently pry the optical element <NUM> away from the holder <NUM> by pressing a tip of the tool into a gap between the peripheral edge of the optical element <NUM> and an inner surface of the holder <NUM>.

As shown in <FIG>, the optical element <NUM> can include, for example, a transparent, translucent, or opaque circular lens <NUM> enclosed by a mounting ring <NUM> sized to be engaged to the holder <NUM>. The optical element <NUM> also includes an obstruction <NUM> over a portion of the lens <NUM> positioned to represent obstructions (i.e., dust, dirt, waste, mud, bird poop, scratches, insects, debris, rain drops, or leaves) that may cover a portion of the field-of-view of the optical sensor <NUM> during real-word use of the vehicle <NUM> and associated cameras and/or sensors <NUM>. For example, the obstruction <NUM> can be formed by adhering a coating, paint, tape, or an adhesive to a portion of the lens <NUM>. Alternatively, obstruction(s) <NUM> can be formed by scratching or otherwise deforming certain portions of the lens <NUM> to distort images captured through the lens <NUM>. In other examples, the optical element <NUM> can comprise an opaque lens <NUM>, which emulates situations when the optical sensor <NUM> is entirely blocked by debris, such that the entire captured image is dark and/or distorted. In other examples, the optical element <NUM> includes a filter <NUM>, such as a neutral-density (ND) filter, as described in further detail herein.

The motorized actuator <NUM> is configured to move the holder <NUM> to adjust the position of the optical element <NUM> relative to the field-of-view of the optical sensor <NUM>. For example, the motorized actuator <NUM> can be configured to move the holder <NUM> between the first or closed position (shown in <FIG>) where the optical element <NUM> is at least partially within the field-of-view of the optical sensor <NUM>, and the second or open position (shown in <FIG>) where the optical element <NUM> is at least partially outside of the field-of-view of the optical sensor <NUM>. For example, the motorized actuator <NUM> can be configured to pivot, swing, rotate, or slide the holder <NUM> away from the housing <NUM>, thereby moving the optical element <NUM> away from the field-of-view of the optical sensor <NUM>.

In some examples, as shown in <FIG> and <FIG>, the motorized actuator <NUM> includes a motor, such as a servomotor <NUM> used, for example, for remote control cars. Suitable servomotors <NUM> that can be adapted for use with the motorized mounting device <NUM> of the present disclosure are widely available from numerous manufacturers including, for example, Kpower Technology Co. , AMain Sports & Hobbies, Savox USA, and others. The servomotor <NUM> can be operatively engaged to the holder <NUM> through linking arms, such as a first arm <NUM> hingedly connected to a second linking arm <NUM>. The servomotor <NUM> is configured to rotate the first arm <NUM>, as shown by arrow A2, about a rotation point <NUM>, which causes the second linking arm <NUM> to move the holder <NUM> (and optical element <NUM> engaged thereto) between the first or closed position (shown in <FIG>) and the second or open position (shown in <FIG>).

With reference to <FIG>, in some examples, multiple motorized mounting devices <NUM> can be connected together through, for example, the receiver box or receiving device <NUM> mounted to the sensor housing or frame <NUM> (as shown in <FIG>), thereby forming a system <NUM> for recording optical sensor <NUM> image data. In some examples, the system <NUM> also includes a power supply <NUM> for providing power for electronics of the receiver box or receiving device <NUM> and/or for providing power for the servomotors <NUM> of the mounting devices <NUM>. The power supply <NUM> can be connected to the receiver box or receiving device <NUM> by a cable <NUM>, wire, or another suitable electrical connection, as are known in the art. In some examples, the power supply <NUM> includes rechargeable or disposable batteries for providing power for the receiver box or receiving device <NUM> and servomotors <NUM>. In other examples, the power supply <NUM> can be integrated with and/or can receive power from an electrical system of the vehicle <NUM>. For example, the power supply <NUM> can include an electrical plug configured to be inserted into and receive power from an auxiliary power outlet or cigarette lighter socket of the vehicle <NUM>.

In some examples, the receiver box or receiving device <NUM> is connected either by wires or wirelessly to the remote control <NUM>. In a simple example, the remote control <NUM> can include a single button or switch <NUM> configured to operate the multiple motorized devices <NUM> simultaneously. When a user presses the button or switch <NUM>, each of the motorized mounting devices <NUM> moves, for example, from the second or open position to the first or closed position and/or from the first or closed position to the second or open position. In other examples, the remote control <NUM> can be configured to allow a user to operate the multiple mounting devices <NUM> independently and/or can be configured to operate the multiple mounting devices <NUM> in sequence. For example, upon pressing a button of the remote control <NUM>, the mounting devices <NUM> may open or close one after another according to a predetermined sequence. In some examples, the optical sensors <NUM> can be connected to a data storage device <NUM> (shown in <FIG>), such as a computer hard drive, for storing images collected by the optical sensors <NUM>. As previously described, the collected image data can be used for testing image analysis software or the vehicle perception system <NUM> to determine whether such software or systems correctly identifies obstructions represented by the optical elements <NUM>.

With reference to <FIG>, in some examples, one or more mounting devices <NUM> can be provided to a user as a kit <NUM> of parts or a set including components needed to install mounting devices <NUM> over an aperture <NUM> of a camera or optical sensor <NUM> of a vehicle <NUM>. An exemplary kit <NUM> of parts including components of the mounting device <NUM> and associated accessories is shown in <FIG>. Specifically, the kit <NUM> can include the motorized mounting devices <NUM>, which each include the holder <NUM>, housing <NUM>, and motorized actuator <NUM>. The provided mounting devices <NUM> can be custom-made devices that are specifically sized for use with a particular camera or optical sensor <NUM>. In other examples, the kit <NUM> can include one or more adapter plates <NUM> configured to be positioned between the device housing <NUM> and the aperture <NUM>, so that the device housing <NUM> can be used with apertures <NUM> of different sizes. For example, the adapter plates <NUM> can be substantially flat annular plates having different inner diameters ID4 that can be positioned between the device housing <NUM> and the sensor housing or frame <NUM>, so that the device housing <NUM> can be used with apertures <NUM> of different sizes. As shown in <FIG>, the kit <NUM> can include multiple adapter plates <NUM> of different common sizes so that the mounting device <NUM> can be used for different sized cameras or optical sensors <NUM>.

The kit <NUM> also includes multiple optical elements <NUM> that can be removably connected to the holder <NUM> of the mounting device <NUM> so that image data including different types of obstructions or distortions can be collected. For example, the kit <NUM> can include optical elements <NUM> including the transparent or translucent lens <NUM> having obstructions <NUM> of different sizes and/or that are positioned over different areas of the lens <NUM>. As previously described, the different obstructions <NUM> can be representative of different objects (i.e., dirt, dust, bird poop, scratches, mud, waste, insects, or rain drops) that may cover a lens of the camera or optical sensor <NUM> during real-world use of the vehicle <NUM> and associated vision sensors. The optical element <NUM> can also include the entirely opaque lens <NUM> that represents times when an entire camera lens is obstructed, such as when a leaf covers the entire optical sensor <NUM> or camera aperture <NUM>. The kit <NUM> can also include the filter <NUM>, such as the neutral-density filter, which limits intensity of light of all wavelengths passing through the filter <NUM>. The neutral-density filter can be configured to approximate low-light conditions, which occur, for example, when the vehicle <NUM> is in a tunnel. Accordingly, when the filter <NUM> is applied, images captured by the optical sensor <NUM> will be dark, at least for a brief time until light levels for the optical sensor <NUM> balance. Removal of the filter emulates exiting a tunnel, which suddenly exposes the optical sensor <NUM> to significantly brighter light (i.e., light of greater intensity). Images captured by the optical sensor <NUM> upon removal of the filter <NUM> will be bright, overexposed, and/or lack contrast, at least until light levels balance.

The kit <NUM> also includes electronic components for operating the mounting device <NUM>. For example, the kit <NUM> can include the receiver box or receiving device <NUM> that is configured to be electrically connected to each of the mounting devices <NUM> by segments of wires <NUM>. The kit <NUM> can also include the remote control <NUM> that is electrically connected to the receiver box by wires <NUM> or by a wireless data connection.

The mounting devices <NUM> of the present disclosure can be used to create datasets of images including images that are at least partially distorted or obscured by the optical elements <NUM>. The created image datasets are used, as previously described, for testing image analysis software or perception systems <NUM> for a vehicle <NUM> to determine whether the software and systems correctly identify the distorted images in the generated test dataset and, if necessary, cause the vehicle <NUM> to take appropriate corrective action. Exemplary perception systems <NUM> and image processing techniques that can be evaluated using the methods disclosed herein are described, for example, in <CIT>, entitled "Methods and systems for lane changes using a multi-corridor representation of local route regions" and <CIT>, entitled "Methods and systems for topological planning in autonomous driving", which are incorporated herein by reference herein in their entireties. A flowchart showing a testing method that can be performed to obtain and analyze a dataset of images is shown in <FIG>.

As shown in <FIG>, at step <NUM>, a user attaches one or more mounting devices <NUM> to the sensor housing or frame <NUM> of the vehicle <NUM>, such that the opening, central bore, and/or interior <NUM> of the device housing <NUM> of each mounting device <NUM> is over the aperture <NUM> of one of the optical sensors <NUM> of the frame <NUM>. For example, the user may attach double-sided tape (e.g., <NUM> VHB tape manufactured by <NUM> Company) to the device housing <NUM> and/or motorized actuator <NUM> of the mounting devices <NUM> and then secure the devices <NUM> to the sensor housing or frame <NUM> proximate to an aperture <NUM> or opening of the optical sensor <NUM>. As previously described, in some examples, the mounting device <NUM> is custom sized for a particular aperture <NUM> or opening. In other examples, the user may attach an adapter plate <NUM> to the device housing <NUM> corresponding to a size of the aperture <NUM> or opening. Setting up the mounting device <NUM> can also include attaching the receiver box or receiving device <NUM> to a convenient location on the exterior surface of the sensor frame <NUM> or vehicle <NUM> and, for example, connecting the mounting devices <NUM> to the receiving box or receiving device <NUM> with the wires <NUM>. In a similar manner, setting up the mounting device <NUM> can also include attaching wires <NUM> of the remote control <NUM> to the receiver box or receiving device <NUM>, so that the mounting devices <NUM> can be operated by the remote control <NUM>.

Once the mounting device or devices <NUM> are in place proximate to the apertures <NUM> of the optical sensors <NUM>, at step <NUM>, image data of objects and/or the environment surrounding the vehicle <NUM> is obtained from the optical sensors <NUM>. During real-world operation of the vehicle <NUM>, image data can be collected as low-resolution data to reduce time required to process or transmit data and/or to reduce computer memory required to store data collected by the optical sensors <NUM>. During real-world operation of the vehicle <NUM> and optical sensors <NUM>, high resolution image data may only be collected, for example, after faults are detected or at other times when collected image data is expected to be particularly relevant, important, and/or interesting. Datasets including the captured low-resolution data, only occasionally interspersed with high-resolution data, may not be suitable for use in testing image analysis software or vehicle perception systems <NUM> because some real-life key events may be missed or only captured at a lower resolution than is needed to make confident coding improvements to the software or perception system <NUM>. Accordingly, entire image datasets generated by the methods disclosed herein can be collected as high-resolution data. The high-resolution data can include both images collected when the optical elements <NUM> are positioned over the optical sensors <NUM> and images collected when the optical elements <NUM> are removed from the field-of-view of the optical sensors <NUM>.

At step <NUM>, the method further includes, as the image data is being captured, causing the mounting device or devices <NUM> to move their respective holders <NUM> to the first or closed position, thereby positioning the optical element <NUM> at least partially within the field-of-view of the optical sensor <NUM>, such that at least a portion of the obtained image data is distorted by the optical elements <NUM>. In some examples, causing the mounting device <NUM> to move the holder <NUM> to position the optical element <NUM> at least partially within the field-of-view of the optical sensor <NUM> occurs manually, actuated by the user. For example, the user can press the button or switch <NUM> on the remote control <NUM> to cause one or more of the mounting devices <NUM> to move the holder <NUM> from the second or open position to the first or closed position, where the optical element <NUM> is in the field-of-view of the optical sensor <NUM>. In other examples, moving the holder <NUM> between the first or closed position and the second or open position occurs automatically. For example, a computer controller associated with the mounting device <NUM>, receiving device or receiving device <NUM>, and/or remote control <NUM> may actuate the mounting devices <NUM> at predetermined times according to a predetermined protocol to obtain a dataset including image distortions at particular predetermined times and/or for predetermined durations.

As previously described, the optical element <NUM> is a camera lens <NUM> including one or more obstructions <NUM> randomly positioned on the lens <NUM> to approximate obstructions that may be present over a camera lens during real-world operation of the vehicle <NUM> and array of optical sensors <NUM>. In such examples, some of the acquired image data can include obscured or disturbed portions caused by the obstructions <NUM>, while other portions of the captured images can appear to be clear. In other examples, the optical element <NUM> can be the opaque lens <NUM>, representing, for example, when a leaf or another object covers the entire aperture <NUM> of the optical sensor <NUM>. In such cases, some portions of captured image data may be entirely dark or distorted.

As previously described, in other examples, the optical element <NUM> is a filter <NUM>, such as the neutral-density filter, that approximates low-light conditions. The neutral-density filter may be used to capture image data that emulates image data that would be obtained when, during real world operation of the vehicle <NUM>, the vehicle is in a dark location, such as a tunnel. In such examples, when the optical element <NUM> comprising the neutral-density filter is in place, the captured image data is dark (i.e., the intensity of light captured by the optical sensor <NUM> is reduced for all wavelengths compared to when no filter <NUM> is present). When a user opens the mounting devices <NUM>, thereby removing the optical element <NUM> including the filter <NUM> from the field-of-view of the optical sensor <NUM>, the obtained image data appears to be exceptionally bright, at least until light levels for the optical sensor <NUM> balance. Desirably, image analysis software should identify such sudden changes in light intensity in captured image data as a normal event that occurs when the vehicle exits a tunnel, and not as an anomaly, malfunction, or other unexpected condition.

At step <NUM>, the method further includes analyzing the obtained image data using the image analysis software or vehicle perception system <NUM> to determine whether the image analysis software or perception system <NUM> correctly identifies portions of the image data that have been distorted by optical element <NUM>. In some examples, image processing and analysis can occur in real time during operation of the vehicle <NUM>. In other examples, image analysis can be a post-processing activity where image data is analyzed at a later time to test, evaluate, and/or optimize the software and perception systems <NUM> or to test new software. For example, different software or algorithms could be used to analyze the same data set, allowing users to assess which software or algorithms are most likely to correctly identify certain image distortions. Generated image datasets can also be used to create simulations including different types of distortions (i.e., with distortions caused by different optical elements <NUM>), so that currently in-use analysis software and systems can be modified and optimized to improve identification for different types of distortions and obstructions.

Claim 1:
A mounting device (<NUM>) for selectively positioning an optical element (<NUM>) within a field-of-view of an optical sensor (<NUM>) of a vehicle, wherein the mounting device (<NUM>) includes a housing (<NUM>) defining an opening (<NUM>) sized to fit over an aperture (<NUM>) of the optical sensor (<NUM>), the mounting device (<NUM>) characterized by:
the optical element (<NUM>) configured to be selectively positioned within a field-of-view of the optical sensor (<NUM>), the optical element (<NUM>) comprising a transparent or translucent lens (<NUM>) and an obstruction (<NUM>) adhered to a portion of the lens (<NUM>);
a holder (<NUM>) for the optical element (<NUM>) connected to the housing (<NUM>) and positioned such that, when the holder (<NUM>) is in a first position, the optical element (<NUM>) is at least partially within the field-of-view of the optical sensor (<NUM>) and the obstruction (<NUM>) of the optical element (<NUM>) blocks a portion of the field-of-view of the optical sensor (<NUM>), and in a second position at least a portion of the field-of-view of the optical sensor (<NUM>) is unobstructed by the holder (<NUM>) and/or by the optical element (<NUM>); and
a motorized actuator (<NUM>) configured to move the holder (<NUM>) between the first position and the second position, thereby at least partially removing distortions of the optical element (<NUM>) from the field-of-view of the optical sensor (<NUM>).