Patent ID: 12243328

DETAILED DESCRIPTION

The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses.

Referring toFIG.1, a road sign interpretation system10includes a camera12provided in or on a vehicle14such as an autonomous vehicle, a battery electric vehicle, a gasoline engine automobile vehicle, a truck, a van, a sport utility vehicle, or the like. The camera12may be any type of color camera and is forward or front-facing to receive a data set having image data16including data defining a roadway18upon which the vehicle14is traveling on and toward, data of one or more other vehicles20also on the roadway18, and data of at least one road sign22providing information to an occupant or operator of the vehicle14related to the roadway18. According to several aspects, the data set of the image data16received by the camera12may be live displayed on a screen24of the camera12or displayed in a similar display provided in the vehicle14.

The road sign interpretation system10utilizes an on-board computer26which is programmed to operate and perform a method of operating the road sign interpretation system10. The on-board computer26receives the image data16noted above from the camera12, as well as other data such as data saved and retrieved from a memory28to perform road sign interpretation and to generate and display data results. The on-board computer26described in reference toFIG.1is programmed to perform operations related to a system and method to collect image data and interpret road signs for the road sign interpretation system10. The on-board computer26is a non-generalized, electronic control device having a preprogrammed digital controller or processor, the memory28or a similar non-transitory computer readable medium used to store data such as control logic, software applications, instructions, computer code, data, lookup tables, etc., and a transceiver or input/output ports. The computer readable medium includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory. The non-transitory computer readable medium excludes wired, wireless, optical, or other communication links that transport transitory electrical or other signals. The non-transitory computer readable medium includes media where data can be permanently stored and media where data can be stored and later overwritten, such as a rewritable optical disc or an erasable memory device. Computer code includes any type of program code, including source code, object code, and executable code. According to other aspects, the computer26may be positioned off-vehicle, including in a remote computing station or in the cloud.

Referring toFIG.2and again toFIG.1, the camera12of the road sign interpretation system10receives the image data16from the camera12. The image data16is compared to common road sign text saved in and retrieved from the memory28of the on-board computer26, including for example road sign text such as “one-way” text30from a road sign32. A fixed size feature vector of the common road sign text saved in the memory28is compared to a fixed size feature vector of text collected by the camera12. Individual fixed size feature vectors define an adjustable hyperparameter that may be varied based on a quantity of sign categories. According to several aspects, the road sign interpretation system10is further able to receive, save and interpret more complex signs and sign data, such as a “road will be closed” text image34presented as a digital image on an electronic, digital sign36, whose text may also be periodically changed as necessary to suit changing roadway conditions, construction and changing conditions affecting the roadway including weather, visibility, and accident conditions.

Referring toFIG.3and again toFIGS.1and2, the system and a method of operation of the road sign interpretation system10is provided using computer vision, natural language text processing (NLP) and planning. The road sign interpretation system10provides for six steps, and the method is performed in six steps. In a first step (1) the image data16discussed above in reference toFIG.1is collected from the front-facing camera12mounted on or in the vehicle14, which provides an egocentric perspective38defining objects visible to the vehicle14as a first-person point of view. The road sign interpretation system10initially detects a generic shape of an exemplary sign40and one or more text instances42defining text tokens of the sign40without any notion of specific sign classes.

In a second step (2) the collected image data16from the front-facing camera12is fed into a first convolutional neural network (CNN)44that yields a set of sign predictions such as one or more sign instances46, and in parallel the image data16collected from the front-facing camera12is also fed into a second CNN48defining a text extractor that extracts text candidates including the text instances42. The text extractor of the second CNN48accepts entire images as input, which locates the text instances42using a fully convolutional neural network as the second CNN48. It is noted the second CNN48is applied as a segmentation network, in lieu of one that yields bounding boxes.

In a third step (3) precise sign and sign data localization is provided for the second CNN48to compute a proper text location50in the text instances42. The text location50in the text instances42is fed from the second CNN48into an optical character recognizer (OCR)52which includes a character recognition algorithm54. Using the character recognition algorithm54the OCR52uses the text location50of the text instances42of one or more digitized forms56that are machine-readable, for example as strings of the text instances42for the computer26to extract. In the OCR52individual text instances42are also constructed to form a model with a long short-term memory (LSTM) architecture to support an ordered sequence of outputs, for example, characters into words. Naive string matching-based approaches will not fit the scalability requirement for the road sign interpretation system10, because it is impractical to enumerate all possible word choices for road signs, therefore in the third step the digitized forms56are taken as an input and an output58defines a specific sign text.

In a fourth step (4) individual sign instances46from the first CNN44and individual ones of the text instances42in the digitized forms56from the OCR52are together fed into a sign text synthesizer module60. The sign text synthesizer module60evaluates a text-sign membership including whether or not sign text lies within a bounding region of one of the sign instances46such as a sign bounding box62. The sign text synthesizer module60also rearranges and configures the detected sign text into a logical reading order, including for example left-to-right and top-to-bottom as individual text instances or as synthesized text instances64. For example, “Physician Parking” and “Doctors only” consist of entirely different strings, but both refer to the same parking guideline/instruction of sign category. To address this challenge, a text processing component is developed that enables semantic understanding of the detected sign text.

To determine text ordering, a first computation determines two-dimensional eigenvectors of individual ones of the text instances42using <x,y> coordination that form segment contours. X-directional eigenvectors are then extended to form line segments, such that line segment endpoints intersect inside the corresponding sign bounding box. If any two-or-more line segments intersect, the corresponding text is appended to a list and reordered by an increasing <x> position, which determines left-to-right text ordering. If multiple text lines exist within a sign instance, they are ordered by an increasing <y> position, which determines top-to-bottom text ordering.

A sign text synthesizer66of the sign text synthesizer module60gathers outputs from the first CNN44, the second CNN48, and the OCR52and synthesizes the outputs into a unifying structure. Specifically, the sign text synthesizer module60determines text-sign membership and governs text ordering such as left-to-right and top-to-bottom. To determine sign-text membership, an overlapping region is computed between the text instances42and the sign bounding boxes62. If an exemplary text instance42is fully encapsulated by the sign bounding box62, the exemplary text instance42is assigned a member of the corresponding sign such as the exemplary sign40.

In a fifth step (5) the text instances42are fed into a semantic encoding and interpretation module68to identify high-level semantics of the detected road signs. Specifically, individual ones of the text instances42are first encoded as fixed-dimension feature vectors70using the semantic encoding and interpretation module68. Subsequently, individual sign categories72are automatically classified as one of the possible road sign categories, for example “no parking” or “detour”, based on the semantics captured as the feature vectors70.

Each detected text instance is first fed into a sentence encoder to obtain a numeric feature representation that captures its semantics. A Universal Sentence Encoder (USE) or any other sentence encoder is leveraged to generate a fixed-length feature vector for each text instance42. Individual text instances42are converted into a data point in a fixed size space. A calculation may then be performed determining how close or distant two points, such as two text instances42are, using any desired distance metrics, for example a Euclidean, or a Dot product. The USE permits measuring a semantic relatedness between text instances, despite them being expressed in different ways.

In a sixth step (6) defining a leveraging and sign context identification step, a planner74defining a portion of a navigation router76receives an output of the semantic encoding and interpretation module68and computes a route plan, which is fed to the navigation router76. Using the route plan the navigation router76recommends a route for the vehicle14to take which optimizes travel time and distance for navigation. The navigation router76communicates the recommended route to a map78which identifies and presents the recommended route with updated path constraints to the vehicle operator for the route recommended to be taken.

Based on the latest sign context, the navigation router76updates the plan and recommends a new route to the vehicle14for navigation guidance including optimizing travel time and distance. Specifically, the navigation router76generates routes from a start point to an endpoint. The navigation router76may for example use Dijkstra's algorithm to route the vehicle14through the map78, which is represented as a graph whose nodes represent intersections and whose edges represent roads.

Referring toFIG.4, a 2D visualization graph80presents individual dots representing text instances and encodings of exemplary categories of signs82collected from 25 sign categories on an exemplary parking sign website, for example 5 samples each, presented with respect to an x-axis84and a y-axis86. The 2D visualization may use a t-Distributed Stochastic Neighbor Embedding (T-SNE) technique that projects the high dimension feature vectors to a 2D space. T-SNE defines a technique or any other dimension reduction technique visualizing high dimensional data by giving each point a location in a two or three-dimensional map. The T-SNE technique is a Stochastic Neighbor Embedding (SNE) variation. Semantically related text instances are localized in the USE encoding space.

Referring toFIG.5, a graph88presents pairwise distances in a semantic encoding space90using a dot product of the USE feature vectors among the 125 sign text instances discussed in reference toFIG.4. Strong within-class similarities are observed, as shown in multiple exemplary squares92along a diagonal94. Given an unseen text instance such as “Smile. You are on Camera”, a USE feature representation is computed and then its nearest neighbor in the encoding space90is identified.

With continuing reference toFIGS.3and5, an output from the semantic encoding and interpretation module68may also be used as an input to the navigation router76. For example, if an encountered sign is classified as “Detour Ahead”, the navigation router76places a blockage ahead of the vehicle14on the map78to remove that path from its routing options.

Referring toFIG.6and again toFIG.5, a location of a new string96is shown in the semantic encoding space90of the 2D visualization graph80. The new string96is near the other text instances in the video surveillance category, despite different wordings and expressions.

Referring toFIG.7, a bipartite graph98presents an illustration of a road segment100which includes a first start point102and a first end point104and a second start point106and a second end point108of the road segment100. The bipartite graph98may be constructed for each side of the road segment100based on the first start point102and the second end point108. The navigation router76can accept blockage locations as inputs to update the bipartite graph98and redirect traffic around those blockages. With continuing reference toFIGS.3and7, the navigation router76may also accept messages that increase a routing cost, or “weight,” associated with a particular edge on the graph98. For example, if the semantic encoding and interpretation module68classifies the sign as “Construction Ahead”, the navigation router76adds an increased routing weight to the road segment100ahead of the vehicle14.

The road sign interpretation system10performs sign detection using a convolutional neural network. Sign detection is performed using a single-stage network, meaning that sign instances such as their locations and classes are extracted by passing an entire image through a single network. This achieves rapid detection, in contrast to traditional techniques, where only individual regions of an image are fed through a network. To achieve scalability the road sign interpretation system10extracts generic sign text and sign text semantics rather than specific sign classes. The system and method used by the road sign interpretation system10is therefore not limited by the kinds of signs detected, but instead extracts sign-like regions, characterized by bounding box locations and a corresponding prediction confidence score.

The road sign interpretation system10of the present disclosure includes a monocular camera and software components that detect and reason about arbitrary road signs to support vehicle driving such as autonomous vehicle driving. The road sign interpretation system10detects road signs in a scalable manner, while also translating its percepts into a set of purposeful actions. The road sign interpretation system10is applicable to autonomously driven vehicles, and enables autonomous vehicles to navigate real-world scenarios, particularly those that are unexpected and challenging, such as construction zones, road closures and accidents.

A road sign interpretation system10of the present disclosure offers several advantages. These include scalable road sign detection to address uncommon or unique road signs. Low-level and high-level interpretation/reasoning capabilities are provided. The system provides the ability to translate visual information into a set of actions. Capabilities for autonomous vehicles to handle uncommon or unexpected situations are also provided as well as improved planning and navigation for autonomous vehicles

The description of the present disclosure is merely exemplary in nature and variations that do not depart from the gist of the present disclosure are intended to be within the scope of the present disclosure. Such variations are not to be regarded as a departure from the spirit and scope of the present disclosure.