Patent Publication Number: US-11645784-B1

Title: Techniques for detecting and classifying relevant changes

Description:
BACKGROUND 
     Technical Field 
     The present disclosure relates generally to change detection, and more specifically to techniques for detecting and classifying relevant changes between three-dimensional (3D) models. 
     Background Information 
     It is increasingly common during the construction and/or operation of infrastructure (e.g., buildings, factories, roads, railways, bridges, electrical and communication networks, equipment, etc.) to create 3D models (e.g., 3D meshes io composed of vertices, edges, and faces). While 3D models may be manually constructed, a number of automated techniques are available that utilize photogrammetry to speed the process. Typically, such photogrammetry-based techniques employ structure-from-motion (SfM) to reconstruct 3D shape based on sets of two-dimensional (2D) source images (e.g., photographs) captured by physical cameras (e.g., digital cameras). 
     Given the ability to automatically generate 3D models, it is possible to efficiently generate a sequence of 3D models that represent a scene including infrastructure at different points in time. Such a sequence may be helpful for a variety of different use-cases, including construction progress monitoring, infrastructure inspection, and mobile mapping, among others. As part of these applications, software may be called upon to detect relevant changes between different 3D models of the sequence, and classify these changes into categories. In this context, the term “relevant” refers to changes that convey information about performance, status, location, or other qualities that are related to the objectives of the use-case. For example, in a construction progress monitoring use-case, changes indicating progress on construction of infrastructure, or the location of raw materials or construction vehicles on the jobsite, may be considered relevant. Conversely, changes indicating natural events (e.g., differing appearance on sunny days vs cloudy days) or peripheral activities (e.g., pedestrians walking about) may be considered irrelevant. 
     In the past, attempts have been made to detect and classify changes between different 3D models by comparing geometry of the 3D models and determining distances (e.g., Euclidean distances) between corresponding objects. If a distance exceeds a threshold, a change may be identified. However, techniques involving distance-based comparisons of geometry have suffered a number of shortcomings. First, these techniques typically detect far too many changes that are not relevant. For example, numerous changes due to natural events, peripheral activities, and the like are often identified that io obscure the relevant changes for the particular use case. Second, these techniques are typically unable to detect changes in color or surrounding context. Changes in color of an object or to what surrounds an object may convey relevant information (e.g., regarding progress, performance, status, etc.) even where the geometry of the object of itself remains unchanged. Third, these techniques are often poorly suited for classifying the is changes they may detect. Filters may be employed to attempt of classify changes identified by through distance-based comparisons of geometry, however such filters typically require a user to explicitly identify a type of object to detect and criteria for determining whether an object falls into that type. In some use cases, however, objects may not have readily identifiable types and explicitly specifying criteria may be exceedingly difficult. 
     There are a number of machine learning techniques that may be used to process 3D models, or data that may be derived from 3D models. Some of these techniques employ only traditional convolutional neural networks (CNNs). Other of these techniques employ sparse lattice networks of bilateral convolution layers (BCLs). Sparce lattice networks take inspiration from permutohedral lattices where convolutions are performed on sparse data in higher-dimensional space. One example of a sparse lattice network is a Sparse Lattice Network for Point Cloud Processing (SplatNet). However, existing CNNs and sparse lattice networks (e.g., SplatNet) have proven inadequate for detecting and classifying relevant changes between 3D models. Among other shortcomings they may require extensive pre-processing which may result in artifacts and loss of natural invariances, may lose some surface information, or may otherwise fail to recognize meaningful information, such that some types of relevant changes are not well detected. 
     Accordingly, there is a need for improved techniques for detecting and classifying relevant changes between 3D models. Further, it would be useful if such techniques had additional application in the wider field of change detection. 
     SUMMARY 
     In various example embodiments, relevant changes between 3D models of a scene including infrastructure at different points in time may be detected and classified by transforming the 3D models into point clouds and applying a deep learning model to the io point clouds. The model may employ a Siamese arrangement of sparse lattice networks, each including a number of modified BCLs. The sparse lattice networks may each take a point cloud as input and extract features in 3D space of the point cloud to provide a primary output with features in 3D space and an intermediate output with features in lattice space. The intermediate output from both sparse lattice networks may be compared is using a lattice convolution layer. The results may then be projected into the 3D space of the point clouds using a slice process and concatenated to the primary outputs of the sparse lattice networks. At this stage, each 3D point in the concatenated output contains information from its point cloud as well as comparison information with respect to other point cloud. Each concatenated output may then be subject to a convolutional network to detect and classify relevant changes. Such techniques may address issues present in existing 3D model change detection techniques. They may also be more widely applicable to change detection between 3D point clouds used for other purposes. 
     In one specific example embodiment, a software application executing on one or more computing devices receives a pair of 3D point clouds of a scene. The software application applies the point clouds to a trained deep learning model, wherein the trained deep learning model includes a Siamese arrangement of sparse lattice networks that each take a point cloud as input and provide a primary output with features in a 3D space of the point clouds and an intermediate output with features in a lattice space used by the sparse lattice networks. The software application infers relevant changes using the trained deep learning model by comparing the intermediate output from both sparse lattice networks to produce a result, projecting the result into the 3D space of the point clouds, concatenating the result to the primary outputs of each of the sparse lattice networks to produce concatenated outputs with both point cloud-specific and comparison information, and applying the concatenated outputs to a convolutional network. The software application outputs an indication of the detected relevant changes. 
     It should be understood that a variety of additional features and alternative embodiments may be implemented other than those discussed in this Summary. This Summary is intended simply as a brief introduction to the reader and does not indicate or io imply that the examples mentioned herein cover all aspects of the disclosure or are necessary or essential aspects of the disclosure. 
    
    
     
       BRIEF DESCRIPTION of the DRAWINGS 
       The description below refers to the accompanying drawings of example embodiments, of which: 
         FIG.  1    is a high-level block diagram of an example software architecture for a software application that may implement the techniques described herein; 
         FIG.  2    is a block diagram of an example architecture of a deep learning model that may be implemented by a change detection process to detect and classify differences in 3D point clouds; 
         FIG.  3 A  is an enlarged block diagram of an example modified BCL that may be used in a modified SplatNet; 
         FIG.  3 B  is a block diagram of an example modified SplatNet that combines a number (e.g., 4) of the modified BCLs of  FIG.  3 A ; 
         FIG.  4    is a block diagram of example lattice convolution sublayers that may be performed by a lattice convolution layer of  FIG.  2   ; 
         FIG.  5    is a block diagram of example slices subprocess that may be performed by slice processes of  FIG.  2   ; 
         FIG.  6    is a flow diagram of an example sequence of steps that may be used by a change detection process to train the deep learning model of  FIG.  2   ; and 
         FIG.  7    is a flow diagram of an example sequence of steps that may be used by a change detection process to use a trained deep learning model of  FIG.  2    to predict relevant changes and categories thereof. 
     
    
    
     DETAILED DESCRIPTION 
       FIG.  1    is a high-level block diagram of an example software architecture for a software application that may implement the techniques described herein. The application may be the ContextCapture™ program available from Bentley Systems, Inc., the Synchro® 4D program available from Bentley Systems, Inc., a combination of programs (e.g., ContextCapture™ and Synchro® 4D software), or other programs developed by the same or other vendors. The architecture may be divided into local software  110  executing on one or more computing devices arranged locally to an end-user (collectively “local devices”), and cloud-based software  112  executing on one or more computing devices is remote from the end-user (“cloud computing devices”), accessible via a network (e.g., the Internet). Each computing device may include processors, memory/storage, a display screen, and other hardware (not shown) for executing software, storing data and/or displaying information. 
     The local software  110  may include frontend clients  120 , operating on local devices. The frontend clients  120  may be of various types, including desktop clients that operate directly under an operating system and web-based clients that operate within a web browser. The frontend clients  120  may, among a variety of other functionality, provide a user-interface to the application. To that end, each frontend client  120  may include a user interface process  122 , among other software processes. 
     The cloud-based software  112  may include backend clients  130  that operate on cloud computing devices. The backend clients  130  may perform, among a variety of other functions, SfM reconstruction to produce 3D models (e.g., 3D meshes composed of vertices, edges, and faces) of scenes including infrastructure from two-dimensional (2D) source images (e.g., photographs) captured by physical cameras (e.g., digital cameras) at different points in time, and 3D change detection to determine and classify relevant changes between pairs of the 3D models associated with different points in time. To that end, each backend client  130  may include a photogrammetry process  132  and/or a change detection process  134 , among other software processes. A services process  140  may functionally organize the application, managing interaction among the clients  120 ,  130  and their processes. 
     It should be understood that while a cloud-based processing architecture  100  is shown, a variety of other software architectures may alternatively be employed. For example, a stand-alone architecture may be employed, where user-interface functionality io and processing-intensive operations are all implemented together on one computing device. 
     In one embodiment, relevant changes between 3D models produced by a photogrammetry process  132  may be detected and classified by a change detection process  134 , by transforming the 3D models into 3D point clouds and applying a deep learning model to the point clouds. The deep learning model may be trained under the direction of, and the results presented via, a user interface process  122 . 
       FIG.  2    is a block diagram of an example architecture of a deep learning model  200  that may be implemented by the change detection process  134  to detect and classify differences in 3D point clouds. A Siamese arrangement is provided where each branch includes a sparse lattice network  220 ,  222 . Each sparse lattice network  220 ,  222  takes as input a point cloud  210 ,  212  (or more specifically, a chunk thereof, as explained further below) corresponding to a different point in time and performs hierarchical and spatially aware learning to produce a primary output  230 ,  232  in 3D space of the point clouds, as well as an intermediate output  234 ,  236  in lattice space of the sparse lattice networks. In one implementation, each sparse lattice network  222 ,  224  is a modified SplatNet that is composed of a number of modified BCLs. BCLs traditionally operate to map input points in 3D space onto a sparse lattice in higher-dimensional space, perform lattice convolutions, and then interpolate the filtered signal back into 3D space. In a traditional SplatNet, the signal in 3D space from a number of BCLs is concatenated together and provided as an output. In a modified SplatNet, in addition to this output, each BCL is modified to provide the filtered signal in lattice space from the lattice convolution. The modified SplatNets provide a plurality of these filtered signals as the intermediate outputs  234 ,  236 . The lattice space of the intermediate outputs  234 ,  236  is common between the two input point clouds  210 ,  212  (or more specifically, chunks thereof), with vertices at similar locations, allowing the exchange of information between them. 
       FIG.  3 A  is an enlarged block diagram  310  of an example modified BCL that may be used in a modified SplatNet. The modified BCL  310  includes three processes: a splat process  320 , a lattice convolution layer  330  and a slice process  340 . In the splat process  320 , the modified BCL projects input features F in 3D space onto a lattice in d 1 —dimensional space defined by lattice features L, via barycentric interpolation. The space between lattice points is controlled by scaling lattice features ΛL, where Λ is a diagonal d 1 ×d 1  scaling matrix. In the lattice convolution layer  330 , once the input points are projected onto the lattice in d 1 —dimensional space, the modified BCL performs a d 1 —dimensional convolution on the splatted signal with learnable filter kernels to produce a is filtered response. This filtered response in lattice space is provided, via path  332 , to the slice process  340 . In addition, the lattice convolution layer  330  is modified to also provide this filtered response in lattice space via path  334 , for inclusion in an intermediate output. In the slice process  340 , the filtered response from path  332  is projected back into 3D space of the point clouds via barycentric interpolation to produce a response in 3D space. 
       FIG.  3 B  is a block diagram  350  of an example modified SplatNet that combines a number (e.g., 4) of the modified BCLs of  FIG.  3 A . The modified SplatNet of  FIG.  3 B  may be used for each sparse lattice network  220 ,  222  in  FIG.  2   . The modified SplatNet begins with a single 1×1 convolutional layer  360 . The 1×1 convolutional layer  360  processes each input point in 3D space separately without any data aggregation. This is followed by a sequence (e.g., 4) BCLs  370 - 376  each operating on a lattice. Each BCL in the sequence operates using a different lattice scale, represented by different Λ. The lattice scale may progress from a given scale Λ 0  (e.g., corresponding to 2 meters) by dividing the lattice scale by a factor of 2, such that Λ 0 ,Λ 1 =Λ 0 /2, Λ 2 =Λ 0 /4 . . . Λ n =Λ 0 / 2 T−1  where T is the number of BCLs. Smaller lattice scale implies coarser lattices and larger receptive fields for the filters. Therefore, deeper BCLs in the sequence have longer range connectivity between input points compared to earlier BCLs. The filtered signal from each modified BCL  370 - 376  in lattice space at each scale Λ 0 -Λ n  is provided as an intermediate output  382 . The response in 3D space from each modified BCL  370 - 376  is provided to a concatenator  390 , and the concatenated result provided as a primary output  392 . 
     Returning to  FIG.  2   , the intermediate outputs  234 ,  236  in lattice space are concatenated by concatenator  240  to aggregate features from the two point clouds  210 ,  212  (or more specifically, chunks thereof). In an implementation where the sparse lattice networks  220 ,  222  are modified SplatNets, the intermediate outputs  234 ,  236  include the io filtered signal in lattice space at each scale Λ 0 -Λ n , so the concatenation includes multiple individual concatenations for each of the lattice scales, such that (signal Λ 0 +signal Λ 0 ) (signal Λ 1 +signal Λ 1 ) . . . (signal Λ n +signal Λ n ). The resulting concatenation of features in lattice space is provided to a lattice convolution layer  250  that compares the features. 
       FIG.  4    is a block diagram of example lattice convolution sublayers  410 - 440  that may be performed by the lattice convolution layer  250  of  FIG.  2   . Each lattice convolution sublayer  410 - 440  receives as input the concatenated signal for the same lattice scale Λ 0 , Λ 1  . . . Λ n . The concatenated features at each lattice scale are then filtered by lattice convolution sublayer  410 - 440  and provided as a series of filtered concatenated outputs in lattice space. 
     Returning to  FIG.  2   , filtered concatenated output  252  from the lattice convolution layer  250  is projected into the 3D space of each of the point clouds using a respective slice process  260 ,  262 . 
       FIG.  5    is a block diagram of example slices subprocess  510 - 540  that may be performed by slice processes  260 ,  262  of  FIG.  2   . Each slice subprocess  510 - 540  receives as input the filtered concatenated signal for the same lattice scale Λ 0 , Λ 1  . . . Λ n . The filtered concatenated features at each lattice scale are then projected back into 3D space of the point clouds via barycentric interpolation. The projections for each lattice scale Λ 0 , Λ 1  . . . Λ n  are then concatenated by concatenator  550  and provided as a comparison output in 3D space. 
     Returning to  FIG.  2   , the comparison output in 3D space of the point clouds from each slice process  260 ,  262  is then concatenated to the primary outputs  230 ,  232  of the sparse lattice networks in 3D by a concatenator  264 ,  266 . After this, each concatenated output includes 3D point information from its point cloud as well as comparison information with respect to the other point cloud. The concatenated output with both point cloud-specific and comparison information is then subject to a convolutional network to detect and classify relevant changes. In one implementation, the convolutional io network includes a 1×1 convolutional layer  270 ,  272  that performs filtering and one or more additional 1×1 convolutional layer  280 ,  282  that predict and classify changes to produce final outputs  290 ,  292 . The final outputs  290 ,  292  may take the form of point clouds (or more specifically, chunks thereof) that indicate (e.g., by highlighting) relevant changes and associate (e.g., with labels or other metadata) each relevant change with a category reflecting the type of change. 
     The weights used in the layers of the deep learning model  200  of  FIG.  2    may be trained by evolving candidate solutions (i.e., sets of weights) to minimize a loss function. One loss function that may be utilized is a categorical cross entropy (CCE) loss function that seeks to minimize the difference between the model&#39;s predicted probability distribution given a training dataset and the actual distribution of probabilities in the training dataset. An example CCE loss function may be given as: 
               C   ⁢   C   ⁢     E   ⁡   (     p   ,   t     )       =     -       ∑     c   =   1     C         t   c     ⁢   log   ⁢     p   c                 
where c is a class in a set of all classes C (e.g., a category of relevant changes of all relevant changes), p c  is the predicted probability of being of the class c, and t c  is the actual probability of being of the class c.
 
       FIG.  6    is a flow diagram of an example sequence of steps  600  that may be used by a change detection process  134  to train the deep learning model  200  of  FIG.  2   . At step  610 , the change detection process  134  receives a training dataset that includes pairs of 3D models that represent scenes that include infrastructure at different points in time. The 3D models in the training dataset may have been reconstructed by a photogrammetry process  132  using SfM from 2D source images captured by physical cameras and then manually annotated to indicate actual relevant changes and categories thereof. Alternatively, the 3D models in the training dataset may have been produced and annotated using other techniques. At step  620 , the pairs of 3D models in the training dataset are transformed into pairs of point clouds (e.g., by point sampling or other techniques). At step  630 , each pair of point clouds is split into a plurality of chunks that each occupy a smaller region of 3D space. The point clouds of a pair may be split upon a same 3D grid or upon other same boundaries, to create a series of pairs of chunks that correspond in size and location with each other. The number of chunks each pair of point clouds is split into may be selected based on available memory resources to avoid memory saturation. At step  640 , data augmentation is applied to increase the size of the training dataset. The data is augmentation may include duplicating pairs of chunks with geometric and/or colorimetric transforms to produce additional chunks that include different combinations of features. At step  650 , features in each chunk of the training dataset are normalized. Normalization may serve to ensure different types of features are given equal (or more equal) importance so that no single type of feature unduly steers model performance. At step  660 , preprocessing is applied to each pair of chunks of the training dataset. The preprocessing may include computing positions, initial weights, and connectivity across points (i.e., filter neighborhoods) for the lattices used in the deep learning model  200 . At step  670 , each preprocessed pair of chunks is applied to the deep learning model  200  and relevant changes and categories thereof are inferred. Such inference is repeated through a series of training iterations, where the predicted relevant changes and categories thereof are compared to the annotations of actual relevant changes and categories thereof, and candidate solutions (i.e., sets of weights) are evolved to minimize a loss function (e.g., a CCE loss function). A gradient descent optimization algorithm may be used to evolve the candidate solutions. Alternatively, another type of optimization algorithm may be used. Training may continue until a stopping condition is reached indicating the loss function has likely been minimized (e.g., norm of the gradient is below a threshold, a max number of iterations has been reached, generalization error begins to increase, etc.). The final candidate solutions (i.e., sets of weights) are then utilized to produce a trained deep learning model  200  that may be used in prediction. 
       FIG.  7    is a flow diagram of an example sequence of steps  700  that may be used by a change detection process  134  to use a trained deep learning model  200  of  FIG.  2    to predict relevant changes and categories thereof. At step  710 , the change detection process  134  receives a pair of 3D models that represent a scene that includes infrastructure at different points in time. The 3D models may have been reconstructed by a photogrammetry process  132  using SfM from 2D source images captured by physical cameras. At step  720 , the pair of 3D models is transformed into a pair of point clouds (e.g., by point sampling or other techniques). At step  730 , the pair of point clouds is split into a plurality of chunks that each occupy a smaller region of 3D space. The point clouds of a pair may be split upon a same 3D grid or upon other same boundaries, to create a series of pairs of chunks that correspond in size and location with each other. The number is of chunks may be selected based on available memory resources to avoid memory saturation. At step  740 , features in each chunk of the pair of point clouds are normalized. At step  750 , preprocessing is applied to each pair of chunks of the pair of point clouds. The preprocessing may include computing positions and connectivity across points (i.e., filter neighborhoods) for the lattice used by the trained deep learning model  200 . At step  760 , each preprocessed pair of chunks of the pair of point clouds is applied to the trained deep learning model  200  and relevant changes and categories thereof are inferred. Relevant changes may be visually indicated (e.g., by highlighting) and categories associated therewith (e.g., by labels or other metadata). At step  770 , the chunks of each point cloud including the relevant changes and categories thereof may be recombined to produce point clouds with classified relevant changes. At step  780 , one or both of the point clouds with classified relevant changes are output (e.g., displayed to a user in a user interface, stored to memory/storage, provided to another software application etc.). 
     It should be understood that a wide variety of adaptations and modifications may be made to the techniques. While specific example software and hardware is discussed above, it should be understood that the techniques may be implemented using a variety of different types of software, hardware, and combination thereof. Such software may include executable instructions stored in a non-transitory computing device-readable medium, such as a volatile or persistent memory device, a hard-disk, or other data storage. Such hardware may include a variety of types of processors, memory chips, programmable logic circuits, application specific integrated circuits, and/or other types of hardware components that support execution of software. Combinations of software and hardware may be adapted to suit different environments and applications. Above all, it should be understood that the above descriptions are meant to be taken only by way of example.