Patent Publication Number: US-11380105-B2

Title: Identification and classification of traffic conflicts

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of and claims priority to U.S. application Ser. No. 15/700,962 filed Sep. 11, 2017, which is a nonprovisional of and claims priority to U.S. Ser. No. 62/393,467 filed Sep. 12, 2016, which is expressly incorporated by reference herein in its entirety. 
    
    
     BACKGROUND 
     Regardless of safety aspects considered by engineers as part of the roadway design and traffic operations, crashes continue to occur. New technologies (e.g., connected vehicles and autonomous vehicles) aiming to simplify driving tasks or remove human factors are expected to dramatically improve highway safety, but likely have implications that are not fully understood until they have been implemented for a certain period of time. For example, vehicle navigation systems help drivers with efficient routing, but pose a distraction concern. Almost any changes in any aspects of roadways, traffic operations and control, vehicle technologies, and driver behaviors will likely have a safety implication. Thus, proactively monitoring and continuous assessment of traffic safety are essential to ensure safe travel environment for all road users. 
     Current practices monitor and evaluate traffic safety based on crashes occurred, which is passive in nature. Historical crash data (post-event data) is typically analyzed at an aggregate (macroscopic) level and generally lacks granular, time-variant information pertaining to each crash occurrence. Because crashes are rare events, this traditional approach is ineffective and cannot be applied in a timely manner, leaving many dangerous locations undetected until after a significant number of crashes have already occurred. Instead of relying on actual crash data, which takes a long time to accumulate, more recent approaches use surrogate measures for crashes, such as “conflict”. Conflict indicates “potential” crash and can be observed in the field by way of avoidance behaviors, such as sudden braking or lane change. 
     Conflicts take much less time to gather than crashes so that safety assessment can be conducted based on conflicts in a timely fashion. It allows for proactive assessment and help to reveal safety concerns or deficiencies before actual crashes occur. 
     Unlike crash data, which is typically entered by a police officer in a crash database during the crash scene investigation, conflict events, being evasive and inconsequent, are rarely reported. Field conflict studies can be conducted by trained personnel; however, doing so is labor intensive and can put observers in dangerous situations. Currently there are no automatic ways to systematically capture and log conflict events. Instead, surrogate safety assessment models have been used to analyze the frequency and characters of narrowly availed vehicle-to-vehicle collisions. Such models allow for an expedited safety assessment, but rely on micro-simulation to predict vehicle trajectories, which is computationally demanding, can be of questionable accuracy, and is not well suited for real-time applications. 
     SUMMARY 
     The inventive disclosed system capitalizes upon richer, conflict event data to effectively improve road safety. It uses applied artificial intelligence, acquired through proper “training” procedures inspired by the fact that a “true” conflict can be easily discerned by human eyes (similar to conducting a field conflict study by trained personnel), to identify and classify various conflict events. After being successfully trained, the system can function like a trained person and is able to continuously and consistently watch for and detect various conflict events exhibited in live images from one or more image capture devices such as cameras that are installed at certain height to monitor live traffic. The existing Closed-Circuit Television (CCTV) cameras, or vehicle detection cameras currently used by public agencies for traffic monitoring, control and management may be used for this purpose. As a result, the system provides an invaluable tool to automatically and continuously monitor and assess traffic safety on roadway networks in real time. By doing so, any potential deficiencies or opportunities for improvement in geometric features, e.g., adding a turn lane, offsetting opposing left turn lanes, flattening horizontal or vertical curves, adding street lighting, etc., or traffic controls, e.g., using protected-only left turn phase, using split phases, adjusting yellow clearance and all-red times, adding advance warning signs, installing raised pavement markers, etc., can be timely discerned and implemented before actual crashes occur. Given its proactive nature, the system is expected to considerably reduce the number of crashes that could have occurred otherwise. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee. 
         FIG. 1  shows an exemplary intersection camera installation, including a CCTV camera for traffic monitoring and a vehicle detection camera for actuated traffic signal operations. 
         FIGS. 2A and 2B  show results of detection and classification of road users using You Only Look Once (YOLO) real time detection system. 
         FIG. 3  is a schematic diagram of an exemplary four-leg signalized intersection and potential conflicts, also showing symbolically represented movements for a conflict between a northeastbound left turn and opposing southwestbound through movement; note the intersection allows a permissive left turn phase. 
         FIG. 4  shows a sequence of images representing the following: (image  1 ) an obtained video image frame with defined region of interest (highlighted field); (image  2 ) a transformed video image frame; (image  3 ) a motion history image showing the history of vehicle positions during the tracking time window); and (image  4 ) a corresponding symbolic image. 
         FIG. 5  shows common vehicle-vehicle conflict types at an intersection. 
         FIG. 6  is a schematic diagram showing the classification of symbolic images (right side) and extraction of “potential” conflict event patterns (left side). 
         FIG. 7  is a representation of an exemplary Convolution Neural Network (CNN) that can be trained with a sequence of input images to detect and classify conflict events with classifications P 1 , P 2  . . . Pn. 
         FIG. 8  depicts an exemplary web-based application interface to the system, which may be implemented using local servers, remote servers, or third party-hosted services (so-called “cloud” providers). 
     
    
    
     DETAILED DESCRIPTION 
     The system obtains live or streaming video feeds from camera(s) as inputs and processes them in real time to identify, classify, and log various traffic conflict events. The camera(s) should be mounted high enough to be able to “see” all road users (e.g., vehicles, bicycles, pedestrians, etc.). Partial or temporary occlusion may be allowed. Existing roadway cameras (CCTV or vehicle detection cameras, such as shown in  FIG. 1 ) could be used for this purpose. The system includes a central processor unit (CPU) together with one or more graphics processing units (GPU) for real-time image processing, a storage device to log detected conflict events, and a communication port to transmit conflict event data to a remote or cloud database server. The processes or steps are as follows: 
     Step  1 . Detect and track moving objects (e.g. vehicles, bicycles, pedestrians, etc.) in the camera view. Detection and tracking are accomplished through known image processing techniques or algorithms (e.g., traditional background subtraction and/or frame difference, or the state-of-the-art You Only Look Once (YOLO) real-time detection system).  FIGS. 2A and 2B  show results of using YOLO for objects (e.g., vehicles) detection and classification based on live images from an existing CCTV camera. The boxes indicate detection. The different colors of boxes indicate different classes of vehicles and the number on each box is the probability or likelihood for that vehicle class. 
     Step  2 . Transform or project the detected moving objects to a two dimensional (2D) top-down view, as shown in  FIG. 3  (Image I-Image  2 ). Some implementations may transform the perspective of a camera view using a mathematical model. Others may use an actual top-down view for the same location from other sources (e.g., high resolution satellite imagery or aerial vehicle mounted camera). For the former, video image frames are georeferenced with respect to four (4) ground reference points, which need to be visible in the camera view. Through those reference points, a perspective transformation model is applied to transform the image coordinate system to a ground coordinate system with respect to the four reference points specified. By way of perspective transformation, the detected objects will be mapped to the 2D top-down view. For the latter, video image frames will generally approximate a 2D plan view, although perspective transformation may still be required for high angle satellite imagery or wide area motion imagery captured from adapted aerial platforms. 
     Step  3 . Vehicle information estimation and correction. Unlike roadway features (which can be deemed as 2D at the ground level), moving objects (e.g., vehicles) are 3D, the images of those objects are 2D projections to the camera view, which vary depending upon the object locations, the actual setup and configuration of the field camera(s) (e.g., location, height, pitch and yaw angles). To identify factual conflicts, the size of vehicles, especially length and width, must be accurately estimated and reflected on the 2D top-down view. To estimate the dimensions of any moving objects (e.g., vehicles) in the original images, a 3D mathematical model corresponding to the camera view is calibrated based on the camera mounting parameters (location [x, y], height [z], zoom, and pitch and yaw angles to replicate the camera view in the field. After calibration, the 3D model is used to estimate the “true” 2D dimensions of objects (e.g., the length and width of vehicles) in the top-down view. Because CCTV cameras are commonly pan-tilt-zoom capable, such 3D model allow for flexibility in tracking with different pan-tilt-zoom settings as long as the camera covers the targeted area of the site. After the dimensions of moving objects are estimated, they are represented as simple shapes (e.g., rectangles) in the 20 top-down view. This view transformation permits accurate capture of headways or gaps between moving objects (e.g., vehicles), which is critical for conflict event identification. 
     Step  4 . Generation of symbolic images. Based on the 2D top-down view images, sequential motion history images for a properly defined time window (e.g., 0.6 seconds) is obtained by sliding the window one frame at a time. A motion history image shows the time sequence of moving objects (vehicles, bicycles, pedestrians, etc.) on a single image frame. An example of such an image is illustrated in  FIG. 4  Image  3 . The information contained in each motion history image is used to detect and classify conflict events. 
     Once the motion history image of moving objects are obtained, they are further reduced or simplified to retain only the necessary information in a pictorial or symbolic way. The actual trajectories of moving objects in a region of interest (ROI) can be constructed as consecutive line segments that connect the center of same vehicles in two successive image frames. The direction of movement may be indicated by an arrow or a circle. The width or thickness of each line segment may be drawn in proportion to its length. By plotting the line segment in this way, the system may provide a visual cue as to whether acceleration or deceleration occurs and to what degree. For example, if the sequence of line segments become thicker along the travel direction, an acceleration is justified. On the other hand, if line segments become thinner along the travel direction, a deceleration is effectual. The overall length of connected line segments in a motion history image implies an average speed over the targeted time window (e.g., 0.6 seconds). Additionally, the priority or order of movements may be indicated by predefined color, gray scale or patterns, in accordance with traffic control (right-of-way rules). An example of resulting symbolic images is illustrated in  FIG. 4  Image  4 . 
     The relative locations of moving objects over the targeted time window, the directions of travel (order of movements), speeds (frame by frame, or average), and acceleration/deceleration characteristics in the symbolic images contain information necessary for detecting and characterizing conflicts events (near-collision situations). By extracting those features from live video images and representing them in a symbolic way on a 2D top-down view, the dimension of original images can be significantly reduced to retain only key features of interest. At the same time, potential private information (e.g., vehicle tag number) is effectively removed for anonymity. To further reduce the dimension of images, several ROIs (see the example in  FIG. 3 ) can be used to segment the images into different targeted areas of conflicts. 
     Step  5 . Identification, classification, and labeling of symbolic images. After the symbolic images (Image  4  in  FIG. 4 ) are generated, unsupervised learning algorithms (such as k-means) are used to classify the symbolic images (e.g., 100 patterns). Normal patterns are expected to occur more frequently on a daily basis, and therefore have a high frequency (probability) of occurrence. “Abnormal” patterns are rare and can be related to conflict events. As such, patterns with low frequency of occurrence may be extracted as representations of “potential” conflict events. The threshold for classifying “low frequency” may be determined from a random process based on an extreme value distribution. Heuristics may be employed to further screen legitimate conflict events. Those extracted “potential” conflict event patterns can then be verified as true conflicts and classified by type.  FIG. 5  shows some vehicle-vehicle conflict types commonly seen at intersections. Verified and classified images are labeled and used to train and test Convolutional Neural Networks (CNN) models. 
     Step  6 . Training and testing of CNN models. The labeled data from Step  5  is used to conduct supervised training of one or more CNN models for identification and classification of conflict events. Training may be conducted on a dedicated high-performance computing machine by using tested algorithms, such as the popular backward propagation algorithm, which adjusts network weights based on computed gradients to minimize an objective (loss) function. CNNs may be initially trained offline using large amounts of historical image data. After being successfully trained and tested, CNNs can be implemented for on-line conflict identification and classification in real time (given steps  1 - 5 ). To reflect any potential changes, such as human behavior, vehicle technologies, etc., CNNs can be retrained with newly acquired data. 
     Step  7 . Optional storage with associated roadway and/or vehicle data. The classified conflict events may be stored in a database containing roadway characteristic data (i.e., number of lanes, approach curvature and grades, posted speeds, etc.), pavement condition data, traffic control data (i.e., pavement marking, signage, signals, etc.), environment data (e.g., weather and lighting conditions), and the like. It will be appreciated that many time-varying characteristics (such as pavement conditions and weather) can be extracted from live images as well. The conflict events identified by CNNs, together with the location, vehicle size, movement direction, speed, deceleration/acceleration, traffic signal operation data (if signalized intersections), and corresponding weather and pavement, may be logged in the database. This conflict event database may then be used to support intersection, corridor, or system-wide analyses for proactive traffic safety management. 
     The system consequently allows for a characterization and representation of true conflict events using “symbolic” features, extracted from live images of CCTV and vehicle detection cameras currently used by public agencies. The system identifies and classifies various conflict events in real time based on a deep-learning approach, allowing for automation of the collection of conflict event data in a continuous and systematic fashion. Thus, it may provide a basis for establishing an active intelligent traffic conflict monitoring and management system that does not currently exist. Such a conflict management system, if implemented, complements existing crash management systems for proactive road safety analysis and management. 
     The embodiments shown and described in the specification are only specific embodiments of inventors who are skilled in the art and are not limiting in any way. Thus, various changes, modifications, and/or alterations to those embodiments may be made without departing from the spirit of the invention or the scope of the following claims.