SYSTEMS, VEHICLES, AND METHODS FOR VEHICLE ENVIRONMENT DETECTION BASED ON BLOCKED VIEW

Embodiments of systems and methods for vehicle environment detection include a vehicle and one or more processors. The vehicle includes a camera operable to generate an image of an environment surrounding the vehicle. The environment includes one or more parking spaces and an object removably attached to the vehicle. The one or more processors are operable to identify the object, generate, using a pre-trained depth algorithm, a depth map based on the image, generate a boundary of the parking spaces based on the depth map excluding the object, determine whether a distance between the boundary of the parking spaces and the vehicle is less than a threshold value, and output an alert in response to determining that the distance is less than the threshold value.

TECHNICAL FIELD

The present specification generally relates to vehicle assistance systems and, more specifically, to vehicle assistance systems using the depth estimation technology.

BACKGROUND

Users may face challenges like limited visibility during parking and reversing, leading to a higher risk of collisions and parking accidents. Obstacles may block the camera or distance sensors of the vehicle to acquire desirable information in understanding the environment around the vehicle. Blind spots and difficulties in judging distances may result in more accidents with other vehicles and pedestrians. Inefficient parking and traffic congestion could occur due to the increased time spent trying to park without guidance systems. Accordingly, there exists a need for detecting the environment around the vehicle using the depth estimation technology even when cameras or sensors of the vehicle are blocked.

SUMMARY

In one embodiment, a system for vehicle environment detection includes a vehicle and one or more processors. The vehicle includes a camera operable to generate an image of an environment surrounding the vehicle. The environment includes one or more parking spaces and an object removably attached to the vehicle. The one or more processors are operable to identify the object, generate, using a pre-trained depth algorithm, a depth map based on the image, generate a boundary of the parking spaces based on the depth map excluding the object, determine whether a distance between the boundary of the parking spaces and the vehicle is less than a threshold value, and output an alert in response to determining that the distance is less than the threshold value.

In another embodiment, a method for vehicle environment detection includes identifying an object removably attached to a vehicle in an image of an environment surrounding the vehicle, the environment including one or more parking spaces, generating, using a pre-trained depth algorithm, a depth map based on the image, generating a boundary of the parking spaces based on the depth map excluding the object, determining whether a distance between the boundary of the parking spaces and the vehicle is less than a threshold value, and outputting an alert in response to determining that the distance is less than the threshold value.

DETAILED DESCRIPTION

Embodiments of systems and methods disclosed herein include a vehicle, one or more cameras, and one or more processors. The cameras are operably imaging an environment around the vehicle. The processor is operable to generate a depth map of the environment based on an image generated by the camera, determine a distance between the vehicle and interested objects in the environment based on the depth map, and cause the vehicle to operate based on the distance according to the interested objects. For example, the system may generate a warning regarding the distance between the vehicle and a parking space or an obstacle. The system may recognize an object in the image attached to the vehicle and determine the distance between the vehicle and the interested objects in the environment by excluding the attached object. The system may recognize parking spaces, determine distances to them, and provide real-time feedback to the driver regarding the distance between the vehicle and available parking spaces, helping to guide parking maneuvers and avoid collisions with nearby objects. Particularly, the system's capability to recognize objects attached to the vehicle and exclude them from distance calculations ensures an accurate assessment of the vehicle's surroundings. This feature prevents false alarms or inaccuracies caused by objects such as bike racks or tow bars, enhancing the system's reliability and usability. By operating the vehicle based on the distances to interested objects, the system can adapt its behavior accordingly, such as adjusting the vehicle speed and applying brakes automatically when approaching obstacles.

As disclosed herein, monocular depth estimation (MDE) refers to a computer vision task regarding predicting the depth information of a scene (e.g., the environment surrounding a vehicle of interest) from one or more images, especially regarding estimating distances of objects in the scene in the one or more images from the viewpoint of the corresponding imaging devices, such as cameras. For example, an MDE algorithm described herein may be a process in computer vision and deep learning where depth information is estimated from one image captured by a single camera. In some embodiments, the MDE algorithm may conduct depth estimation based on multi-view geometry of rectified stereo- or multi-camera images. The MDE algorithms described herein may include machine-learning functions to predict depth from the images. The MDE algorithms may include depth and pose networks, where the depth network predicts depth maps of the scene, and the pose network estimates the camera's motion between successive frames. Accordingly, by reconstructing the 3D structure of the scene from images, the MDE-based techniques described herein enhance the understanding of the vehicle's surrounding environment for obstacle avoidance, scene reconstruction, and object recognition.

Referring now to figures,FIG.1depicts a visual sonar system100. The visual sonar system100may include one or more cameras110(for example, a front camera110a, one or more side cameras110b, and a rear camera110c) attached to a vehicle101and configured to image an environment111surrounding the vehicle in real-time. The cameras110may be operably generating one or more images301(e.g., as illustrated inFIGS.3A and3B) of the environment111around the vehicle101. The environment111may include one or more obstacles307and one or more parking spaces305. The vehicle101may include one or more processors132(e.g., as illustrated inFIG.2). The processors132may perform MDE to generate a depth map401(e.g., as illustrated inFIGS.4A and4B) of the environment111based on the images301(e.g., as illustrated inFIGS.3A and3B) generated by the cameras110to evaluate the distances and relative positions between the vehicle101, the parking spaces305, and the obstacles307in the environment111.

The vehicle101may be an automobile or any other passenger or non-passenger vehicle such as, for example, a terrestrial, aquatic, and/or airborne vehicle. The vehicle101may be an autonomous vehicle or a semi-autonomous vehicle that navigates its environment111with limited human input or without human input. The vehicle101may include actuators for driving the vehicle, such as a motor, an engine, or any other powertrain. The vehicle101may move or appear on various surfaces, such as, without limitation, roads, highways, streets, expressway, bridges, tunnels, parking lots, garages, off-road trails, railroads, or any surfaces where the vehicles may operate. For example, the vehicles101may move within a parking lot or parking place, which includes one or more parking spaces305. The vehicle101may move forward or backward.

As mentioned above, the vehicle101may include one or more cameras110. The cameras110may be mounted to the exterior of the vehicle101at the front of the vehicle101, at the rear of the vehicle101, on the side of the vehicle101, on top of the vehicle101, and/or at any other location on the vehicle101. For example, the cameras110can be mounted to the rear of the vehicle101and/or one or more side view mirrors of the vehicle101and can have a field of view of various objects in the environment111, such as an attached object121mounted on the vehicle101. The cameras110may be, without limitation, one or more of monocular cameras, red-green-blue (RGB) cameras, or red-green-blue-depth (RGB-D) cameras. The cameras110may be configured to capture one or more images301of the environment111. The images301may be, without limitation, monocular images, RGB images, or RGB-D images. The one or more processors132may generate one or more depth maps401based on the images301, where the pixel values of the depth map401may be proportional to the distance between the cameras110and the attached objects121in the image.

The visual sonar system100may include one or more vehicle modules, which include one or more machine-learning algorithms, such as a depth algorithm. The depth algorithm may be an MDE algorithm. The visual sonar system100may generate, using the depth algorithm, depth maps of objects in one or more images301captured by the cameras110. In some embodiments, the depth algorithm may conduct a depth estimation using stereo vision techniques, which may rely on two or more cameras of the cameras110to calculate depth by triangulation. In some other embodiments, the depth algorithm may estimate depth using images taken by a single camera of the cameras110, such as the MDE-based technologies.

In embodiments, the depth algorithm, such as the MDE algorithm, may use models to generate depth maps401, including, without limitation, Convolutional Neural Networks (CNNs) to learn hierarchical features from images for spatial information estimation, Recurrent Convolutional Neural Networks (RNNs), such as Long Short-Term Memory (LSTM) networks, to capture temporal dependencies in sequential data, Encoder-Decoder Architectures, such as U-Net, to extract features from the images301to generate the corresponding depth maps401, Residual Networks (ResNets), such as ResNet-50 and ResNet-101, to address the vanishing gradient problem for improved depth estimation performance, and Generative Adversarial Networks (GANs) to generate realistic depth maps by learning the distribution of depth information in training data and producing high-quality depth estimations for single images.

In some embodiments, one or more attached objects121may be attached to the vehicle101. The objects may be, without limitation, a cargo, a trailer, a bicycle, a kayak, a canoe, a surfboard, a paddleboard, a toolbox, camping gears, a ladder, an emergency light, or any objects suitable to be attached to the vehicle. The vehicle101may include one or more attachment accessories120, configured to moveably attach or mount the attached objects121to the vehicle101. The attachment accessories120may include, without limitation, a stand, a rack, a cargo carrier, a roof rack, a bed extender, a tow hook, a tow strip, a hitch receiver, a suction cup, a magnetic mount, a customized welding or fabrication, or any combination thereof.

The attached objects121attached to the vehicle101and the attachment accessories120may be imaged by the one or more cameras110and included in the imaged environment111around the vehicle101in one or more images301. The images301may be, without limitation, monocular images, RGB images, or RGB-D images. When the visual sonar system100generates a depth map401of the environment111based on an image301generated by the camera110, the depth map401may include a boundary of the parking space305and a boundary of the vehicle101. The boundary of the vehicle101may include the vehicle101, the attached objects121, and the attachment accessories120.

FIG.2is a schematic showing the various systems of the vehicle101. It is to be understood that the vehicle101is not limited to the systems and features shown inFIG.2and that each may include additional features and systems. The vehicle101may be an automobile, a boat, a plane, or any other transportation equipment. The vehicle101may also or instead be a device that may be placed onboard an automobile, a boat, a plane, or any other transportation equipment. As shown, the vehicle101may include a data unit118for generating, processing, and transmitting data.

The data unit118includes an electronic control unit (ECU)108, a network interface hardware106, one or more imaging sensors104, such as cameras110, a screen122, a navigation module124, a speaker125, and one or more motion sensors136that may be connected by a communication path126. The network interface hardware106may connect the vehicle101to external systems via an external connection128. For example, the network interface hardware106may connect the vehicle101to other vehicles directly (e.g., a direct connection to another vehicle proximate to the vehicle101) or to an external network such as a cloud server.

Still referring toFIG.2, the ECU108may be any device or combination of components including one or more processors132and one or more non-transitory processor-readable memory modules134. The one or more processors132may be any device capable of executing a processor-readable instruction set stored in the one or more non-transitory processor-readable memory module134s. Accordingly, the one or more processors132may be an electric controller, an integrated circuit, a microchip, a computer, or any other computing device. The one or more processors132is communicatively coupled to the other components of the data unit118by the communication path126. Accordingly, the communication path126may communicatively couple any number of processors132with one another, and allow the components coupled to the communication path126to operate in a distributed computing environment. Specifically, each of the components may operate as a node that may send and/or receive data.

The one or more non-transitory processor-readable memory modules134may be coupled to the communication path126and communicatively coupled to the one or more processors132. The one or more non-transitory processor-readable memory modules134may include RAM, ROM, flash memories, hard drives, or any non-transitory memory device capable of storing machine-readable instructions such that the machine-readable instructions can be accessed and executed by the one or more processors132. The machine-readable instruction set may include logic or algorithm(s) written in any programming language of any generation (e.g., 1GL, 2GL, 3GL, 4GL, or 5GL) such as, for example, machine language that may be directly executed by the one or more processors132, or assembly language, object oriented programming (OOP), scripting languages, microcode, etc., that may be compiled or assembled into machine readable instructions and stored in the one or more non-transitory processor-readable memory modules134. Alternatively, the machine-readable instruction set may be written in a hardware description language (HDL), such as logic implemented via either a field programmable gate array (FPGA) configuration or an application-specific integrated circuit (ASIC), or their equivalents. Accordingly, the functionality described herein may be implemented in any conventional computer programming language, as pre-programmed hardware elements, or as a combination of hardware and software components. In embodiments, the one or more non-transitory processor-readable memory modules134may store one or more vehicle modules, one or more machine-learning algorithms, and one or more depth algorithms.

In embodiments, the ECU108may conduct the MDE using the depth algorithms. The depth algorithms may be pre-trained using sample images and depth maps. The vehicle modules may be trained and provided with machine-learning capabilities via a neural network as described herein. By way of example, and not as a limitation, the neural network may utilize one or more artificial neural networks (ANNs). In ANNs, connections between nodes may form a directed acyclic graph (DAG). ANNs may include node inputs, one or more hidden activation layers, and node outputs, and may be utilized with activation functions in the one or more hidden activation layers such as a linear function, a step function, logistic (Sigmoid) function, a tanh function, a rectified linear unit (ReLu) function, or combinations thereof. ANNs are trained by applying such activation functions to training data sets to determine an optimized solution from adjustable weights and biases applied to nodes within the hidden activation layers to generate one or more outputs as the optimized solution with a minimized error. In machine learning applications, new inputs may be provided (such as the generated one or more outputs) to the ANN model as training data to continue to improve accuracy and minimize error of the ANN model. The one or more ANN models may utilize one-to-one, one-to-many, many-to-one, and/or many-to-many (e.g., sequence-to-sequence) sequence modeling. The one or more ANN models may employ a combination of artificial intelligence techniques, such as, but not limited to, Deep Learning, Random Forest Classifiers, Feature extraction from audio, images, clustering algorithms, or combinations thereof. In some embodiments, a convolutional neural network (CNN) may be utilized. For example, a convolutional neural network (CNN) may be used as an ANN that, in the field of machine learning, for example, is a class of deep, feed-forward ANNs applied for audio analysis of the recordings. CNNs may be shift or space-invariant and utilize shared-weight architecture and translation. Further, each of the various modules may include a generative artificial intelligence algorithm. The generative artificial intelligence algorithm may include a general adversarial network (GAN) that has two networks, a generator model and a discriminator model. The generative artificial intelligence algorithm may also be based on variation autoencoder (VAE) or transformer-based models. For example, the depth algorithm may involve training convolutional neural networks (CNNs) on large datasets containing pairs of example images and their corresponding depth maps. The depth maps provide ground truth depth information for each pixel in the example images. The CNN may learn to map input example images to corresponding depth maps by capturing the spatial relationships between objects and their depths in the example images.

Still referring toFIG.2, one or more imaging sensors104, such as cameras110, are coupled to the communication path126and communicatively coupled to the one or more processors132. While the particular embodiment depicted inFIG.2shows an icon with one imaging sensor104and reference is made herein to “imaging sensor” in the singular with respect to the data unit118, it is to be understood that this is merely a representation and embodiments of the system may include one or more imaging sensors104having one or more of the specific characteristics described herein.

The one or more imaging sensors104may include one or more cameras110, such as the front camera110a, the side cameras110b, and the rear camera110c. The one or more cameras110may be, without limitation, one or more of monocular cameras, RGB cameras, or RGB-D cameras. The cameras110may be, without limitation, one or more of rearview cameras, side-view cameras, front-view cameras, or top-mounted cameras. In some embodiments, the one or more imaging sensors104may be any device having an array of sensing devices capable of detecting radiation in an ultraviolet wavelength band, a visible light wavelength band, or an infrared wavelength band. The one or more imaging sensors104, such as the cameras110, may have any resolution. In some embodiments, one or more optical components, such as a mirror, fish-eye lens, or any other type of lens may be optically coupled to the one or more imaging sensors104. In embodiments described herein, the one or more imaging sensors104may provide image data to the ECU108or another component communicatively coupled to the communication path126. The image data may include image data of the environment111around the vehicle101. In some embodiments, for example, in embodiments in which the vehicle101is an autonomous or semi-autonomous vehicle, the one or more imaging sensors104may also provide navigation support. That is, data captured by the one or more imaging sensors104may be used by the navigation module124to autonomously or semi-autonomously navigate the vehicle101.

The one or more imaging sensors104, such as the cameras110, may operate in the visual and/or infrared spectrum to sense visual and/or infrared light. Additionally, while the particular embodiments described herein are described with respect hardware for sensing light in the visual and/or infrared spectrum, it is to be understood that other types of sensors are contemplated. For example, the systems described herein could include one or more LIDAR sensors, radar sensors, sonar sensors, or other types of sensors and such data could be integrated into or supplement the data collection described herein to develop a fuller real-time traffic image.

In operation, the one or more imaging sensors104, such as the cameras110, capture image data and communicate the image data to the ECU108and/or to other systems communicatively coupled to the communication path126. The image data may be received by the processor132, which may process the image data using one or more image processing algorithms. The imaging processing algorithms may include, without limitation, an object recognition algorithm, such as a real-time object detection models, and a depth algorithm, such as the MDE depth algorithm. Any known or yet-to-be developed video and image processing algorithms may be applied to the image data in order to identify an item or situation. Example video and image processing algorithms include, but are not limited to, kernel-based tracking (such as, for example, mean-shift tracking) and contour processing algorithms. In general, video and image processing algorithms may detect objects and movements from sequential or individual frames of image data. One or more object recognition algorithms may be applied to the image data to extract objects and determine their relative locations to each other. Any known or yet-to-be-developed object recognition algorithms may be used to extract the objects or even optical characters and images from the image data. Example object recognition algorithms include, but are not limited to, scale-invariant feature transform (“SIFT”), speeded-up robust features (“SURF”), and edge-detection algorithms. The image processing algorithms may include machine learning functions and be trained with sample images including ground truth objects and depth information.

The network interface hardware106may be coupled to the communication path126and communicatively coupled to the ECU108. The network interface hardware106may be any device capable of transmitting and/or receiving data with external vehicles or servers directly or via a network. Accordingly, network interface hardware106can include a communication transceiver for sending and/or receiving any wired or wireless communication. For example, the network interface hardware106may include an antenna, a modem, LAN port, Wi-Fi card, WiMax card, mobile communications hardware, near-field communication hardware, satellite communication hardware and/or any wired or wireless hardware for communicating with other networks and/or devices. In embodiments, network interface hardware106may include hardware configured to operate in accordance with the Bluetooth wireless communication protocol and may include a Bluetooth send/receive module for sending and receiving Bluetooth communications.

In embodiments, the data unit118may include one or more motion sensors136for detecting and measuring motion and changes in motion of the vehicle101. Each of the one or more motion sensors136is coupled to the communication path126and communicatively coupled to the one or more processors132. The one or more motion sensors136may include inertial measurement units. Each of the one or more motion sensors136may include one or more accelerometers and one or more gyroscopes. Each of the one or more motion sensors136transforms the sensed physical movement of the vehicle101into a signal indicative of an orientation, a rotation, a velocity, or an acceleration of the vehicle101. In some embodiments, the one or more motion sensors136may include one or more steering sensors. The one or more steering sensors may include, without limitation, one or more of steering angle sensors, vehicle speed sensors, gyroscopes, inertial measurement units, or any other steering sensors operable to collect data on vehicle trajectory. For example, the steering angle sensor may measure the rotation of the steering wheels of the vehicle101and provide data on the angle at which the steering wheel is turned, indicating the intended direction of the vehicle. The vehicle speed sensors may monitor the speed of the vehicle wheels to provide real-time data on the vehicle's speed. The gyroscopes may detect the changes in orientation and angular velocity of the vehicle101by measuring the rate of rotation around different axes.

In embodiments, the data unit118includes a screen122for providing visual output such as, for example, maps, navigation, entertainment, seat arrangements, real-time images/videos of surroundings, or a combination thereof. The screen122may be located on the head unit of the vehicle101such that a driver of the vehicle101may see the screen122while seated in the driver's seat. The screen122is coupled to the communication path126. Accordingly, the communication path126communicatively couples the screen122to other modules of the data unit118. The screen122may include any medium capable of transmitting an optical output such as, for example, a cathode ray tube, a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a liquid crystal display, a plasma display, or the like. In embodiments, the screen122may be a touchscreen that, in addition to visually displaying information, detects the presence and location of a tactile input upon a surface of or adjacent to the screen122. The screen may display images captured by the one or more imaging sensors104, such as the cameras110. In some embodiments, the screen may display a depth map that is generated based on the image captured by the one or more imaging sensors104, such as the cameras110.

In embodiments, the data unit118may include the navigation module124. The navigation module124may be configured to obtain and update positional information of the vehicle101and to display such information to one or more users of the vehicle101. The navigation module124may be able to obtain and update positional information based on geographical coordinates (e.g., latitudes and longitudes), or via electronic navigation where the navigation module124electronically receives positional information through satellites. In certain embodiments, the navigation module124may include a GPS system.

In embodiments, the data unit118includes the speaker125for transforming data signals into mechanical vibrations, such as in order to output audible prompts or audible information to a driver of the vehicle. The speaker125is coupled to the communication path126and communicatively coupled to the one or more processors132. The speaker125may output a warning sound based on distances between the vehicle101and external objects measured by the visual sonar system100.

In embodiments, the one or more processors132may operably control the steering and break of the vehicle101to enable the vehicle101to perform various maneuvers, such as, without limitation, accelerating or decelerating to reach a desirable velocity, stopping at desirable position, and turning at desirable angle.

Referring now toFIGS.3A-4B, example images301captured by cameras110and example depth maps401generated by the visual sonar system100are depicted. In embodiments, the one or more cameras110of the vehicle101may image the environment111surrounding the vehicle101to generate one or more images301. As illustrated inFIGS.3A and3B, in some embodiment, the images301may include one or more parking space305and one or more obstacles307. Each parking space305may include, without limitation, a parking stall, markings, symbols (e.g., no parking zones, accessible parking designations, loading/unloading areas), wheel stops303, signage (e.g., parking regulations, time limits, permit requirements, restrictions, safety warnings), or other structure and elements associated with the parking space305. One or more objects and obstacles307may be present near or around the parking spaces305, such as the wheel stop303, and physical structures such as walls or barriers as part of the parking building. The obstacles307may be positioned close to parking spaces in a way that drivers need to be mindful of their proximity to the obstacles307when maneuvering into or out of parking spaces. The obstacles307may be marked with caution stripes or hazard stripes. In some embodiments, as illustrated inFIG.3A, the image301may not include any attached object121or attachment accessory120. In these embodiments, the visual sonar system100may not use the images301to identify any attached object121and to generate a boundary of the parking spaces305excluding the attached object121.

As illustrated inFIG.3B, in some embodiments, the image301taken by the cameras110, such as the rear camera110c, may include, the obstacles307, the parking space305, the wheel stop303, one or more attached objects121, such as a bike attached to a rack attached to the vehicle101. The attached objects121may block partial or full views of some of the environment111surrounding the vehicle of the images301. For example, the bike as the attached object121blocks the partial view of the parking spaces305, and the wheel stops303.

In some embodiments, the visual sonar system100may use the images301to identify the attached object121and to generate a boundary of the parking spaces305excluding the attached object121. The visual sonar system100may use real-time object detection models, such as, without limitation, YOLO, and Faster R-CNN to identify the attached object121, such as a bike. The real-time object detection models may be pre-trained by using annotated sample images captured from different viewpoints around one or more sample vehicles, which may be the same model as the vehicle101, where the annotated images may include the boundaries of the sample vehicles and/or one or more sample attached objects, and the training process involving backpropagation and optimization of the real-time object detection models to minimize the difference between predicted and ground truth bounding boxes of the sample attached objects. In some embodiments, the visual sonar system100may use the depth maps401generated based on the images301to identify the attached objects121, as further described in the following paragraphs. In some embodiments, the visual sonar system100may use side cameras110bto determine the boundary of the attached objects121.

As illustrated inFIGS.4A and4B, the visual sonar system100may generate the depth map401based on the images301. For example, the system may use one or more of the depth algorithms, such as the MDE algorithms, to generate depth maps401from the input images301. The visual sonar system100may extract relevant features in the images301using machine-learning functions, such as CNNs to capture desired visual cues. The visual sonar system100may then process these features using a depth prediction network that learns to map the features to depth values. The visual sonar system100may estimate the distances of objects, such as the parking spaces305, the obstacles307, and the attached objects121, in the environment111surrounding the vehicle101from the viewpoint of the camera110(e.g., the rear camera110c) capturing the image301. For example, as illustrated inFIG.4A, the depth map401is generated based on the image301inFIG.3A, where no attached object121is captured within the image301. The shapes, locations, and depth information of the objects, such as the obstacles307and the wheel stops303, are represented in the depth map401, with the dark monochromatic color representing near and light monochromatic color representing far to the rear camera110c. Similarly, as illustrated inFIG.4B, the depth map401is generated based on the image301inFIG.3B, which includes the bike as the attached object121is attached to the rear of the vehicle101. The attached object121in the depth map401has the darkest color suggesting the attached object121is the closest object to the rear camera110c. The attached object121, however, may block the partial or full view of the obstacles307and the wheel stops303.

The visual sonar system100may recognize the attached object121based on the image301ofFIG.3B. In some embodiment, the visual sonar system100may recognize the attached object121using the one or more pre-trained real-time object detection models, as discussed further above. In some embodiments, the visual sonar system100may recognize the attached object121based on the depth map401. For example, in some embodiments, the visual sonar system100may identify the attached object121from the image301based on a comparison of depths in the depth map401and an attachment depth threshold. For example, the attachment depth threshold may represent a maximum allowable distance between the vehicle101and the attached object121in the depth map401. The visual sonar system100may determine that objects within the attachment depth threshold are part of the vehicle101and are not treated as obstacles307during maneuvers, such as moving backward. The attachment depth threshold may be set based on the physical dimensions of the vehicle101, the precision of depth sensing technology of the vehicle101, and the expected range of distances between the vehicle101and any attached components. The attachment depth threshold may be manually changed by the user.

In some embodiments, the cameras110may continuously generate the images301in a sequence of time frames. The visual sonar system100may generate corresponding depth maps401from the images301in the sequence of time frames. The visual sonar system100may identify the attached object121from the corresponding depth maps401representing a substantially constant depth and a substantially constant coordinate in the corresponding depth maps401. In some embodiments, when the cameras110continuously generates images301in the sequence of time frames, the vehicle101may further using the one or more steering sensors to generate a real-time trajectory of the vehicle101. The trajectory may represent the path or movement of the vehicle101over time, such as trajectory information of the vehicle's position, orientation, velocity, and acceleration. The visual sonar system100may identify the attached object121based on the relative motion of the attached object121against the vehicle101and the real-time trajectory of the vehicle101. By comparing the relative motion of the attached object121in the images301and/or the depth maps401against the vehicle trajectory, the visual sonar system100may identify the attached object121that exhibits motion patterns consistent with being attached to the vehicle101.

In some embodiments, the system may determine whether one or more of the images are suitable for an environment detection purpose based on the blockage percentage of the objects attached to the vehicle. For example, the visual sonar system100may determine a blocking percent based on the attached object121relative to the parking space305in the image301or the depth map401. The visual sonar system100may determine whether the blocking percent is greater than a block threshold, and in response to determining that the blocking percent is greater than the block threshold, the visual sonar system100may not use the depth map401to determine whether the vehicle101is too close to the obstacles307and output an undesired condition alert. The block threshold may be 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%, or any value between 10% and 100%. For example, when the visual sonar system100determines that more than 50% of the view in the image301or the depth map401is blocked, the visual sonar system100may output a message indicating failure for environment detection due to attached object blockage (e.g., determining a distance between a boundary of the parking space305and the vehicle101). In some embodiments, the visual sonar system100may use images captured by another camera110for the environment detection.

In embodiments, the depth algorithms, such as the MDE algorithms, may be pre-trained. The visual sonar system100may train the depth algorithms on datasets with ground truth images and corresponding depth maps. The visual sonar system100may optimize the models in the depth algorithms for depth map predictions through validation processes, such as backpropagation. The visual sonar system100may further apply post-processing to refine the depth map to output the depth map as a grayscale image representing estimated object distances to the cameras taking the image. For example, the pre-training may include labeling the example images and desirable depth information in the images and using one or more neural networks to learn to predict the desirable and undesirable depth information from the input images based on the training data. The pre-training may further include fine-tuning, evaluation, and testing steps. The vehicle modules of the depth algorithms may be continuously trained using the real-world collected data to adapt to changing conditions and factors and improve the performance over time. The neural network may be trained based on the backpropagation using activation functions. For example, the encoder may generate encoded input data h=(Wx+b) that is transformed from the input data of one or more input channels. The encoded input data of one of the input channels may be represented as hij=g(Wxij+b) from the raw input data xij, which is then used to reconstruct the output {tilde over (x)}ij=f(Wxij+b′). The neural networks may reconstruct outputs, such as the depth information in the depth map, into x′=(WTh+b′), where W is weight, b is bias, WT, andbare transverse values of W and b and are learned through backpropagation. In this operation, the neural networks may calculate, for each input data, the distance between an input data x and a reconstructed input data x′, to yield a distance vector |x-x′|. The neural networks may minimize the loss function which is a utility function as the sum of all distance vectors. The accuracy of the predicted output may be evaluated by satisfying a preset value, such as a preset accuracy and area under the curve (AUC) value computed using an output score from the activation function (e.g. the Softmax function or the Sigmoid function). For example, the visual sonar system100may assign the preset value of the AUC with a value of 0.7 to 0.8 as an acceptable simulation, 0.8 to 0.9 as an excellent simulation, or more than 0.9 as an outstanding simulation. After the training satisfies the preset value, the pre-trained or updated depth algorithm may be stored in the ECU108. In embodiments, the visual sonar system100may conduct a similar pre-training process to the real-time object detection models with ground truth input images and corresponding identified objects, particularly with the ground truth input images captured by vehicle onboard cameras and the corresponding identified objects being attached to the sample vehicles.

The visual sonar system100may generate a boundary of the parking spaces305based on the depth map401excluding the attached object121. In embodiments, the visual sonar system100may generate the depth map401without the attached object121after recognizing the attached object121. Based on the continuously captured images301, the visual sonar system100may further generate an extended boundary of the parking space305by aggregating space and time information in the corresponding depth maps401generated from the continuously captured images301. The boundary and the extended boundary of the parking space305may be two-dimensional or three-dimensional. The visual sonar system100may continuously determine whether the distance between the boundary of the parking spaces305and the vehicle101is less than a threshold value, and output an alert in response to determining that the distance is less than the threshold value. The threshold value may be predetermined based on, without limitation, the dimensional of the vehicle101, the reaction time of a user, and/or the velocity of the vehicle101. In some embodiments, the visual sonar system100may operate the vehicle101to avoid a collision between the vehicle101and the parking space305or the obstacles307in response to determining that the distance is less than the threshold value.

Referring toFIG.5, the operations of the visual sonar system100described herein are depicted. The process of the operation may be divided into two parts. In the first part, the images301generated by the cameras110(such as the front camera110a, the side cameras110b, and the rear camera110c) are processed separately. Note that the images being processed as illustrated inFIG.5may be generated by a single camera110but at different time stamps. In some embodiments, the images may be generated by different cameras110at the same time stamp or at different time stamps. For each image such as IMG1and IMG N inFIG.5, the image may be processed into an MDE image in step501. In step502, the visual sonar system100may calculate the three-dimensional (3D) input of the image. In step503, the visual sonar system100may conduct a ground removal of the image. In step504, the system may conduct a scale estimate of the image. In the second part, the processed images may be aggregated in step521using point aggregation technology such that the generated aggregated images include both time and space information of the environment111.

Referring toFIG.6, a flowchart of illustrative steps for vehicle environment detection based on the blocked view of the present disclosure is depicted. At block601, the method600for vehicle environment detection includes identifying an attached object121removably attached to the vehicle101in the image301of an environment111surrounding the vehicle101. The environment111includes one or more parking spaces305and the attached object121. At block602, the method600includes generating, using a pre-trained depth algorithm, the depth map401based on the image301. At block603, the method600includes generating a boundary of the parking spaces305based on the depth map401excluding the attached object121. At block604, the method600includes determining whether a distance between the boundary of the parking spaces305and the vehicle101is less than a threshold value. At block605, the method600includes outputting an alert in response to determining that the distance is less than the threshold value.

In some embodiments, the attached object121may be identified from the image301or the depth map401. For example, the method600may further include identifying the attached object121using one or more pre-trained real-time object detection models. In some embodiments, the method600may further include identifying the attached object121based on a comparison of depths in the depth map and an attachment depth threshold. In some embodiments, the method600may further include continuously generating the images301captured in a sequence of time frames, generating corresponding depth maps401in the sequence of time frames, identifying the attached object from the corresponding depth maps401representing a substantially constant depth and a substantially constant coordinate in the corresponding depth maps.

In some embodiments, the method600may further include continuously generating images301captured in a sequence of time frames, generating corresponding depth maps401in the sequence of time frames, and generating an extended boundary of the parking spaces305by aggregating space and time information in the corresponding depth maps401.

In some embodiments, the method600may further include generating a real-time trajectory of the vehicle101using one or more steering sensors of the vehicle101, continuously generating images301captured in a sequence of time frames, and identifying the attached object121based on a relative motion of the attached object121against the vehicle101and the real-time trajectory of the vehicle101. The one or more steering sensors may include, without limitation, a steering angle sensor, a vehicle speed sensor, a gyroscope, or a combination thereof.

In some embodiments, the method600may further include generating a blocking percent based on the attached object121relative to the parking spaces305in the image301or the depth map401, determining whether the blocking percent is greater than a block threshold, and in response to determining that the blocking percent is greater than the block threshold, outputting an undesired condition alert for determining the distance between the boundary of the parking spaces305and the vehicle101.

In some embodiments, the method600may further include operating the vehicle101to avoid a collision between the vehicle101and the parking spaces305or the obstacle307in response to determining that the distance is less than the threshold value. The parking spaces305may include, without limitation, a parking stall, markings, wheel stops, or a combination thereof.

It is to be understood that the embodiments are not limited in their application to the details of construction and the arrangement of components set forth in the description or illustrated in the drawings. The invention is capable of some embodiments and of being practiced or of being carried out in various ways. Unless limited otherwise, the terms “connected,” “coupled,” “in communication with,” and “mounted,” and variations thereof herein are used broadly and encompass direct and indirect connections, couplings, and mountings. In addition, the terms “connected” and “coupled” and variations thereof are not restricted to physical or mechanical connections or couplings.