Patent Description:
Construction projects can involve hundreds of people during the build phase - including manufacturers, suppliers and installers for a number of different trades associated with the construction project.

Construction projects include a tendering process whereby interested parties tender for job(s) within the construction project. The tender process generally requires interested parties to provide a detailed, itemised list of all materials that need to be manufactured/supplied/installed as part of the project. This list is known as the Bill of Materials (BOM).

In order for interested parties to tender, two-dimensional (2D) drawings of the construction project (generally in PDF format) are provided. From those 2D drawings, a BOM is manually created by one or more individuals analysing the drawings, and identifying the types/quantities of materials used. This process is commonly known as "material take-off".

Material take-off may be carried out by manually measuring, counting, and colouring in all materials on a 2D drawing (often on a paper print out of the PDF). For a large-scale construction project, this may take interested parties involved in the tender process upwards of two days of full-time labour (as well as utilising the services of an experienced estimator). As will be appreciated, the manual nature of the process creates delays as well as enormous risk for miscalculations, which can result in cost overruns during the construction phase.

Attempts have been made to automate material take-off. For example, there exists software that receives architectural 3D models of a construction project (e.g. buildings or the like), which allows users to select pieces of material, and associate them with particular product types. This allows a user to export a list of all materials required for the project, and to track them individually in the 3D model. A drawback of this type of software is that material take-off is still extremely labour intensive and also requires specialised computing skills to use the software. To date, these types of software systems have had extremely low take-up by installation/manufacturing companies.

There also exists software tools that allow users to "digitally" perform manual estimating activities from 2D drawings, such as ticking off/colouring in individual materials to count the total number, or to use a "digital measuring tape" to measure dimensions on the images. This process is almost as manual as traditional pen-and-paper methods (often taking days per project for just a single type of material i.e. glass), and creates a lot of risk of human-error.

It would be desirable to provide a method and system which ameliorates or at least alleviates one or more of the above mentioned problems or provides a useful alternative.

<NPL>, relates to symbol recognition to take image-based engineering diagrams or hand sketches and provide engineering models by way of Convolutional Neural Networks to provide a trainable engineering symbol recognizer where the Convolutional Neural Networks is applied either using a multi-scale sliding window or to isolated pixel clusters obtained through Connected Component Analysis (CCA).

<NPL>, relates to digitizing and extracting data from piping and instrumentation diagrams which can contain noisy or irrelevant information, and utilises deep learning to identify and isolate pipeline codes, pipelines, inlets and outlets, and to detecting symbols from these diagrams and to thereby create a tree like data structure of the piping schematics.

<NPL>, discloses a system for recognizing symbols and connection lines in drawings such as flowcharts, logic and electrical circuits, and chemical plant diagrams by way of (<NUM>) rules, which are used to segment symbols from connection lines in the drawing image, the image being thinned, vectorized, and pre-processed in routine ways and (<NUM>) a drawing understanding subsystem in concert with the rules (<NUM>) to classify symbols and correct errors.

A reference herein to a patent document or other matter which is given as prior art is not to be taken as an admission that that document or matter was known or that the information it contains was part of the common general knowledge as at the priority date of any of the claims.

According to a first aspect, the present invention provides, a system for determining material take-off from a 2D drawing, the system including: a pre-processing component operable to receive and pre-process one or more 2D drawings to provide one or more processed images; a categoriser component operable to receive the processed image from the pre-processing component, the categoriser component including one or more pre trained convolutional neural networks, the categoriser component operable to determine the type of the processed image from one or more categories of drawing types in order to match category of drawing-types with the appropriate one or more pre-trained neural networks; a material identifier component operable to receive the category of drawing-types, and determine the suitable pre-trained neural network to apply to the processed image, and provide a multi-dimension matrix of values associated with the processed image wherein each value in the multi-dimension matrix represents the probability that a feature in the processed image is present and to generate one or more multi-dimension probability matrix (MPMs) for the processed image; an MPM decoding component operable to decode the one or more MPMs generated by the material identifier component to produce one or more data objects for each feature found in the processed image; and an output component operable to provide one or more of: a unique identifier for each feature; a list of coordinates indicating the location of the feature on the processed image; and/or a list of coordinates describing the location of any text or other encoded information that is associated with the feature.

Preferably, the pre-processing component is further operable to convert the 2D drawing to one or more of: a predetermined format, size and aspect ratio. The 2D drawing may take any format and for example may be one or more of a pdf, jpg, dwg file. Preferably, the size of the <NUM> x <NUM> pixels but it will be appreciated that any size could be used depending on the application.

Preferably, the pre-processing component further includes an image rescaling component operable to normalise the processed image.

It will be appreciated that the one or more convolutional neural networks may include an input layer of predetermined dimensions. The input layer may be, for example, <NUM> x <NUM> x <NUM> layers.

The one or more convolutional neural networks may include one or more of convolutional layers containing one or more nodes, the one or more nodes each having one or more weights and biases. The one or more convolutional layers may also correspond to the number of supported drawing types.

Preferably, the material identifier component includes one or more pre-trained material identifying neural networks. The one or more pre-trained material identifying neural networks are preferably trained to produce a multi-dimensional matrix of values.

Preferably, the MPM represents one or more of the numbers, types, physical location and dimension of each feature associated with the processed image; and the MPM being encoded in the values assigned to each X and Y pixel coordinate on the drawing.

It will be appreciated that the feature may take any form and could include one or more of a material, parts of the structure including walls & rooms, furniture or other elements visible on the drawings.

Preferably, the MPM decoding component is operable to scan each coordinate represented in the MPM and to determine if one or more coordinates in the processed image contains one or more of: (a) a material; (b) no material; or (c) the edge of a new material. The MPM decoding component may be further operable to scan adjacent coordinates and check the values for each adjacent coordinate thereby determining borders and/or associated text or other property types which are represented by the MPM.

Preferably, the system further includes a post-processing component operable to perform checks on the data to improve operation of the system. The post-processing component may include an OCR subsystem component operable to runs an optical character recognition process over the coordinate locations associated with the features which were identified by the MPM.

Preferably, the post-processing component further includes a quality assurance subsystem component operable to provide a user a review of the output by the MPM decoding component. The quality assurance subsystem component provides an interactive processed image where coordinates for each feature identified on the drawing are used to render highlighting on the features for easy of identification. The quality assurance subsystem component may include the BOM for the drawing rendered in a table which can be edited by a user such that new features may be added to the BOM table if they were omitted by the system.

Preferably, the quality assurance subsystem component includes a draw/drag/erase tool that allows the user to create/modify/delete coordinates on the processed image.

Preferably the system further includes a training data component which receives the 2D drawings together with the generated BOMs via the MPM decoder; the 2D drawings together with the generated BOMs via the MPM decoder being fed back into a training data set for the current features.

According to a second aspect, the present invention provides, a method for determining material take-off from a 2D drawing, the method including the steps of: receiving and pre-processing one or more 2D drawings to provide one or more processed images; determining the type of the processed image from one or more categories of drawing types by way of one or more pre trained convolutional neural networks and determine the type of the processed image from one or more categories of drawing types in order to match category of drawing-types with the appropriate one or more pre-trained neural networks; receiving the category of drawing-types, and determining the pre-trained neural network to apply to the processed image; providing a multi-dimension matrix of values associated with the processed image wherein each value in the multi-dimension matrix represents the probability that a feature in the processed image is present; generating one or more multi-dimension probability matrix (MPMs) for the processed image; decoding the one or more MPMs produce one or more data objects for each feature found in the processed image; and outputting one or more of: a unique identifier for each feature; a list of coordinates indicating the location of the feature on the processed image; and/or a list of coordinates describing the location of any text or other encoded information that is associated with the feature.

The invention will now be described in further detail by reference to the accompanying drawings. It is to be understood that the particularity of the drawings does not superseded the generality of the preceding description of the invention.

The present invention may be utilised by users working in the construction sector, and it will be convenient to describe the invention in relation to that exemplary, but non-limiting, application. It will be appreciated that the present invention is not limited to that application and may for example, be applied in electronics fabrication, clothing design & manufacture, or anywhere that design drawings are currently analysed by humans.

Referring to <FIG>, there is shown a diagram a system <NUM> for automated material take-off with devices making up the system, in accordance with an exemplary embodiment of the present invention. The system <NUM> includes one or more servers <NUM> which include one or more database <NUM> and one or more computing devices <NUM> (associated with a user for example) communicatively coupled to a cloud computing environment <NUM>, "the cloud" and interconnected via a network <NUM> such as the internet or a mobile communications network.

Although "cloud" has many connotations, according to embodiments described herein, the term includes a set of network services that are capable of being used remotely over a network, and the method described herein may be implemented as a set of instructions stored in a memory and executed by a cloud computing platform. The software application may provide a service to one or more servers <NUM>, or support other software applications provided by a third party servers. Examples of services include a website, a database, software as a service, or other web services. Computing devices <NUM> may include smartphones, tablets, laptop computers, desktop computers, server computers, among other forms of computer systems.

The transfer of information and/or data over the network <NUM> can be achieved using wired communications means or wireless communications means. It will be appreciated that embodiments of the invention may be realised over different networks, such as a MAN (metropolitan area network), WAN (wide area network) or LAN (local area network). Also, embodiments need not take place over a network, and the method steps could occur entirely on a client or server processing system.

Referring now to <FIG>, there is shown a flowchart illustrating the process steps <NUM> adopted by the system and method for automated material take-off in accordance with an exemplary embodiment of the present invention. The method begins at step <NUM> where a user associated with, for example the computing device <NUM> of <FIG> provides 2D drawings which may be uploaded to the server <NUM> and stored on database <NUM> in cloud <NUM>.

The 2D drawings may be provided by the user in any number of formats including pdf, jpg, dwg and the like. Control then moves to step <NUM> where the 2D drawings are provided to a pre-trained neural network. The pre-trained neural network may exist on server <NUM> or database <NUM> and may be pre-trained using a traditional, supervised approach. For example sample construction drawings of a specific type may be provided (i.e. shop drawings for pieces of aluminium cladding, elevations, electrical designs, etc.) as well as human-generated examples of data-abstractions of the BOM for example. As will be described further with reference to <FIG>, the pre-trained neural network may take any number of forms. The pre-trained neural network at step <NUM> processes the data from the 2D drawings. Control then moves to step <NUM> where the neural network generates an abstraction of the BOM.

At steps <NUM> and <NUM> at each step of the training process for training the neural network, each layer of a machine learning algorithm receives an input and provides an output based on weights and biases associated with each node in the layer. The final layer produces a generated multi-dimensional probability matrix (MPM) which is compared against the human created examples of MPM, as will be appreciated by those skilled in the art. The difference between the machine generated and human generated MPMs provides a loss function which allows the weights and biases to be updated for all nodes in the network, such that future machine generated MPMs will be increasingly similar to the human generated version. By way of many thousands of generations of training across hundreds of examples of training data machine generated MPMs will converge on the human generated versions providing the ability to generate a BOM without human oversight.

Once the MPM has been produced, control moves to step <NUM> where an output component processes the MPM into a human-readable BOM.

Advantageously, other than providing the 2D drawings the user requires, no interaction with the system is required and the user associated with computing device <NUM> and receives the result which is the BOM that they can then use as part of their tender process or the like or to order the materials they require knowing that the BOM is accurate.

<FIG> is a flow chart <NUM> illustrating the process steps adopted by the system and method for automated material take-off in accordance with a further exemplary embodiment of the present invention. Control starts at step <NUM> where the user associated with computing device <NUM> uploads one or more 2D drawings to the cloud <NUM> which contains the server <NUM> and database <NUM> where the system <NUM> may process the 2D drawing. Upload step <NUM> may be carried out via a simple web-based application which allows the user associated with the computing device <NUM> to upload files to a cloud-based computing system <NUM> capable of performing the calculations and storing both files and metadata. All subsequent operations may then be performed in the cloud <NUM>. The cloud <NUM> may include both virtual and physical computers networked together, may connect two or more databases for the purposes of sharing data between network computers and may have access to networked storage locations for the purposes of storing/sharing files between components.

Control then moves to step <NUM>, <NUM> and <NUM> which may be considered to be part of a pre-processing component. At step <NUM> the uploaded file from step <NUM> is converted to a format which is a standard size and aspect ratio. The drawings may take the form of pdf, jpg, dwg or the like but the conversion step uses a <NUM> x <NUM> pixel image size. It will be appreciated that any size may be provided. The conversion is due to the neural-network requiring a consistent size for all images it processes. Control then moves to step <NUM> where a categoriser component contains a pre-trained convolutional neural network, which may be built using the methods known to the skilled person to provide:.

As will be appreciated, the purpose of the categorising component is to determine which type of 2D construction drawing has been uploaded by the user in order to provide specific processors depending on the drawing type. For example, a particular type of 2D drawing may show only the structural steel components for a project, where another one shows the ventilation or plumbing.

This is due to the fact that a single feature (i.e. a door) appears differently depending on the type of drawing it appears. For example a technical specification of a door construction will look different to a floorplan which will look different again to a forward-facing "elevation" drawing on the front of a building. Advantageously, the categorising component makes this determination automatically. This is done by way of training the network on a pre-categorised list of thousands of drawings based on which features they contain and whether they show the side, the front, internals, or overhead view of a building, or whether they show a detailed/component level of specific features (i.e. in the case of a feature being one or more materials, this could include cladding, structural steel, modular bathrooms fixtures and the like). Advantageously, this aspect of the invention obviates the need for a user to select which type of drawing they are uploading but rather simply upload the drawing.

Control then moves to step <NUM> where an image rescaling component normalises the images of the 2D drawings for processing. Preferably the drawings are converted to a <NUM> x <NUM> pixel image but it will be apparent that any image resolution may also be applicable depending on the application. Control then moves to the material identifier component <NUM> which is a family of pre-trained material identifying neural networks. Each of these neural networks are trained to receive a single 2D drawing image of a specific type (i.e. elevations, floorplans, etc.) and for a particular type of feature (i.e. a particular material or structure, for example, a window), the neural network is trained to produce a multi-dimension matrix of values, where each value represents the probability that a particular feature of a given feature (i.e. the edge of a window or the centre of a piece of steel) appears and gives coordinates set in the 2D drawing. Each of these sets of outputs may be denoted as a MPM.

At step <NUM> the MPM takes the form of a <NUM> x <NUM> x <NUM> x <NUM> array of integers which represents the pixel coordinates of the input data (<NUM> x <NUM> with <NUM> x <NUM>) representing the probability that specific trained features of the features are present in the current pixel of the 2D drawing.

In one embodiment some of the features encoded into MPM are as follows:.

As will be appreciated, each of the above are distinct values from one another and for example, if the pixel in location <NUM> on the X axis and <NUM> on the Y axis represents the centre of the feature this may be recorded such that [<NUM>, <NUM>] equals [<NUM>, <NUM>, <NUM>].

This is just one example of an encoding of what would be tens or hundreds of thousands and it would be appreciated that any number of encodings could be created, representing different properties of the features on a 2D drawing. For example, using different values to encode corners, common symbols or curves and the like. It will be appreciated that the specific encodings may vary drastically between features and drawings types and are relatively arbitrary provided they capture of the pertinent information for features on a given drawing type.

Advantageously the MPM of a 2D drawing represents an abstracted encoding of a BOM in that the numbers, types, physical location and dimension of each feature on the drawing are encoded in the values assigned to each X and Y pixel coordinate on the drawing.

The neural networks in the material identifier component <NUM> may be trained by providing pre-constructed MPMs. In this way, the material identifier component <NUM> "learns" to generate its own MPMs based on previously unknown types of construction drawings. The values in the MPMs generated by the material identifier component reflect the probability of particular features being in a specific pixel location - for instance, using the encoding above, [<NUM>,<NUM>] = (<NUM>,<NUM>,<NUM>), may represent that the material identifier component <NUM> has a roughly <NUM>% confidence that the specific pixel represents the centre of a feature.

In this implementation, the individual neural network models within the material identifier component <NUM> may include a common type of machine learning algorithm, known as adversarial networks. These networks will be familiar to a skilled person and consist of a "generator" and a "discriminator", which are themselves two separate machine learning networks.

As will be understood to a skilled person, the generator is a machine learning algorithm which receives an input (in this case, a 2D drawing), and uses this to generate novel examples of the desired output (in this case the MPM). This is then passed to the discriminator, along with a pre-prepared version of the ideal output (the "correct" MPM), which must choose which is the "real" output. Two loss functions are calculated, such that, whenever the discriminator successfully detects the "incorrect" MPM created by the generator, the discriminator receives positive reinforcement, and the generator does not. Alternatively, if the discriminator is "fooled" into selecting the MPM created by the generator, then the generator receives the positive reinforcement, and the discriminator does not.

As will be appreciated, over time, this leads to the generator becoming increasingly skilled at creating outputs that match the human-created, ideal MPMs for a particular 2D drawing, while the discriminator becomes ever-better at identifying the human-created ones from the "fakes". Ultimately, this results in the generator becoming so good, that the MPMs it generates are almost indistinguishable from the human-created ones. Advantageously, given MPMs represent an abstract encoding of the BOM, this means the material identifier component <NUM> is ultimately able to "read" a 2D drawing, and output a list of all features present in the drawing, including dimensions, categories, and other pertinent information present in the training data set.

Control then moves to step <NUM> where an MPM decoding component decodes the MPM generated by a material identifier component to produce a simple data object for each feature found in the 2D drawing. This process involves scanning through each coordinate represented in the MPM data and performing a simple check to determine if it contains a feature, no feature or the edge of a new feature and the like. Once the new features are located in the MPM, the MPM decoder scans adjacent coordinates and checks the values for each adjacent coordinate in a way that is allows it determine the borders and associated text or other property types which are represented by the MPM.

For each feature located in this matter, the details captured are stored by the MPM decoding component. The MPM decoding component for each of the one or more feature data objects being outputted provides:.

Control then moves to post-processing components <NUM>, <NUM> and <NUM> which perform checks on the data and/or improves the overall system <NUM>.

At step <NUM> an Optical Character Recognition (OCR) subsystem component runs an optical character recognition algorithm over the coordinate locations associated with the individual features which were generated at step <NUM> and <NUM>, any text identified by the OCR subsystem component may then be stored with the feature data object for ultimate display to the user, which will be described further below. The OCR subsystem component may be utilised to check text which appears on the drawing, but preferably it is further operable to associate that text with the features located in the drawing based on their location in the drawing. For example, the OCR subsystem component may "read" the widths of a feature which appears on the drawings thereby knowing which feature that width applies to.

Control then moves to step <NUM> in which a quality assurance subsystem component provides an opportunity for an individual user or a group of users to review the output from step <NUM>, that is the BOM data against each of the 2D drawings to verify their integrity before providing the finally BOM to the end user associated with computing device <NUM>. The Quality Assurance (QA) subsystem component may be, for example, a web based application which allows the user to be presented with:.

Advantageously, the user can edit or correct and that edit/correction is learned by the system, providing improved accuracy and the like over time.

Control then moves to step <NUM> which is a training data component which may receive as input, 2D drawings which were provided at step <NUM> together with BOMs which have been output via the MPM decoder at step <NUM> which are then fed back into the training data set for the current materials identifier at step <NUM>. In this way, the materials identifier component is provided with an ever-increasing set of training data allowing it to learn from mistakes identified by the QA subsystem at step <NUM>. Otherwise, control moves to step <NUM> in which an output component provides the completed BOM to the end user associated with computing device <NUM>.

In this embodiment, the BOM is provided to the user in the form of an interactive web application which allows then to search/view/edit any individual feature listed in the BOM. The user may additionally be provided with an interactive version of their 2D drawings where the coordinates identified in the BOM allow highlights of the features to be digitally rendered over the 2D drawing. The BOM may be provided in any number of formats to the end user as will be appreciated by those skilled in the art.

Where the terms "comprise", "comprises", "comprised" or "comprising" are used in this specification (including the claims) they are to be interpreted as specifying the presence of the stated features, integers, steps or components, but not precluding the presence of one or more other features, integers, steps or components, or group thereof.

Claim 1:
A system for determining material take-off from a 2D drawing, the system including:
a pre-processing component operable to receive and pre-process one or more 2D drawings to provide one or more processed images;
a categoriser component operable to receive the processed image from the pre-processing component, the categoriser component including one or more pre-trained convolutional neural networks, the categoriser component operable to determine the type of the processed image from one or more categories of drawing types in order to match category of drawing-types with appropriate one or more pre-trained material identifying neural networks;
a material identifier component operable to receive the category of drawing-types, and determine suitable pre-trained material identifying neural network to apply to the processed image, and generate a multi-dimension matrix of values associated with the processed image wherein each value in the multi-dimension matrix represents the probability that a feature in the processed image is present and to generate one or more of such a multi-dimension matrix as one or more multi-dimension probability matrices denoted MPMs for the processed image;
an MPM decoding component operable to decode the one or more MPMs generated by the material identifier component to produce one or more data objects for each feature found in the processed image; and
an output component operable to provide one or more of: a unique identifier for each feature; a list of coordinates indicating the location of the feature on the processed image; and/or a list of coordinates describing the location of any text or other encoded information that is associated with the feature.