# Automatic Transformation of HVAC Diagrams into Machine-Readable Format

Noah Mertens, Tommy Wohlfahrt, Nick Hartmann, Chethan Reddy

► To cite this version:

Noah Mertens, Tommy Wohlfahrt, Nick Hartmann, Chethan Reddy. Automatic Transformation of HVAC Diagrams into Machine-Readable Format. 19th IFIP International Conference on Product Lifecycle Management (PLM), Jul 2022, Grenoble, France. pp.410-419, 10.1007/978-3-031-25182-5_40. hal-05420955

HAL Id: hal-05420955

https://inria.hal.science/hal-05420955v1

Submitted on 17 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Automatic Transformation of HVAC Diagrams into Machine-Readable Format

Noah Mertens(✉)D, Tommy WohlfahrtD, Nick HartmannD, and Chethan Babu Venkata ReddyD

Fraunhofer IIS EAS, Dresden, Germany
noah.mertens@eas.iis.fraunhofer.de
https://www.eas.iis.fraunhofer.de/

**Abstract.** Accounting for around 40% of total energy consumption, buildings in industrialized countries are a particular focus for saving energy. Especially digital twins and building information modeling (BIM) can serve as a mean to improve operational processes, detect faulty ventilation and air conditioning (HVAC) components which lead to an increased energy consumption and evaluate future renovation possibilities during a buildings life cycle. To enable these advantages a transition from existing HVAC diagrams to a machine-readable representation including spatial and functional interrelationship of all components as well as their semantic relationship is necessary. Especially in retrofit cases the manual transition is a time-intensive and an error prone process. The present work aims to develop a procedure for the (partially) automated recognition of HVAC diagrams and extraction of information about their intercorrelation. In order to achieve the desired results, we use multiple approaches of computer vision, both conventional as well as more recent ones using artificial intelligence such as “Faster R-CNN”. The developed pipeline consists of three basic steps: Find symbols of components, find connecting lines, combine the extracted information into a machine-readable representation. Due to privacy issues, only very few real diagrams are available. To evaluate our approach we developed a data generator to train and test our pipeline. The obtained results show that to a large extent it is already possible to transfer relevant information from technical diagrams into a machine-readable format in order to reduce the effort of creating and validating digital twins and BIM for retrofits.

**Keywords:** Ontology · HVAC diagram recognition · Machine learning · Computer vision · Building information modelling

## 1 Introduction and Problem

In industrialized countries the building sector is accountable for around 40% of the total energy consumption. Therefore it becomes a particular focus for saving energy and reducing its environmental impacts. In order to attain a more

© IFIP International Federation for Information Processing 2023
Published by Springer Nature Switzerland AG 2023
F. Noël et al. (Eds.): PLM 2022, IFIP AICT 667, pp. 410–419, 2023.
https://doi.org/10.1007/978-3-031-25182-5_40
efficient use of resources [12], cost and time reduction [11] during a buildings lifecycle, multiple stakeholders like architecture, engineering, construction and facility management (FM) depend on the exchange of information across organizational boundaries. Especially digital twins and building information modeling (BIM) can serve as a mean to improve operational processes, detect faulty HVAC components which lead to an increased energy consumption and evaluate future renovation possibilities during a buildings life cycle. To enable these advantages, a transition from existing HVAC diagrams to a machine-readable representation including spatial and functional interrelationship of all components as well as their semantic relationship is necessary. Creating digital twins and implementing BIM into the daily workflow is a difficult challenge, especially in retro fit cases where documentation exists mostly in a non-machine-readable format.

The present work aims to develop a procedure for the (partially) automated recognition of HVAC diagrams and extraction of information about their intercorrelation for real world applications. In previous research on this topic a multi-step approach was developed [8]. It contains the recognition of only four types of symbols (damper, heat exchanger, pump, valve), as well as the recognition of connecting lines and a basic identification of line intersections. Although it is a promising approach, it lacks important characteristics to apply for real world use-cases. Especially the number of symbols, but also the capability and robustness of line and intersection detection are the main limiting factors. Also commonly seen problems with noise such as manually added text, folding lines or dirt on scanned images are not taken into consideration. Even though, it created a starting point for the development of our pipeline, which mainly focuses on the improvement of robustness of the overall process and applicability for real use-cases.

Our developed pipeline consists of three basic steps, which are discussed more detailed in Sect. 4:

1. Find symbols of components

2. Find connecting lines (including intersections)

3. Combine the extracted information into a machine-readable representation

Our approach could reduce the effort for creation and validation of digital twins and BIM significantly by providing a promising starting point for real world applications as well as further research towards a fully automated process.

At the time of this research we could not identify a suitable public data set containing HVAC diagrams. In order to develop and evaluate our procedure we created a comprehensive data generator, which is described in Sect. 3. It includes multiple kind of symbols, combinations of components and line intersections as well as different kind noise. In Sect. 4 our approach is described in detail and the achieved results are discussed in Sect. 5.

## 2 Machine-Readable Representation

A digital representation of the HVAC diagram contains all relevant technical components, as well as their connections. Even though most existing semantic
ontologies in the building sector put their focus on different goals [5], they are all based on the idea of graph structures. They consist of three concepts: Classes, relationship and attributes. To allow a flexible deployment of the extracted information for streamline applications we decided to use a basic graph approach. A conversion to more domain-specific ontology or API-friendly formats for further usage like creating a digital twin or implementing BIM is easily possible. Furthermore, by remaining on a high-level graph structure we avoid limitations of more specific ontologies such as limited types of components or attributes.

Graphs capture interactions (edges) between units (nodes) which allows storing and accessing relational knowledge about interacting entities [2]. The graph represents all relevant components as nodes and their connecting lines as edges. The developed process for line detection provides a robust way to describe the structure unequivocally. As of now the graph is undirected and therefore contains no information about the direction of flow inside the system.

# 3 Data Generator

## 3.1 Data Structure

Due to privacy aspects, the number of publicly available HVAC diagrams is limited. For the purpose of evaluation and testing of our approach a data generator was created. The most common components of HVAC systems are grouped to a pool which can be extended with new symbols at any time. Out of this pool, the generator chooses randomly symbols and connection line colors to generate the diagram. The aim of the data generation is the optimization and evaluation of the overall approach, hence the technical usefulness of the generated diagrams are not always given. Therefore it not only creates the diagram, but also the corresponding ground truth information. The generation can be divided into the following steps.

**Basic Shape.** A $NxM$ grid structure forms the base graph of every diagram, where a node represents a component and the edges represent the connection lines between them (cf. Fig. 1a). The size and shape of the graph are freely selectable.

**Removing Edges.** A parameter determines the number of randomly selected edges (connecting lines) to be removed in the graph (cf. Fig. 1b). Therefore not all nodes contain exactly 4 connections and thus represent a more realistic diagram structure. If there are several completely separated graphs after this step, only the graph with the most nodes is used for the further procedure.

**Create Connection Nodes.** In order to realize L-, T- and 4-way intersections in the diagram, a parameter determines how many nodes represent only a intersection instead of a component. In the case of a 4-way intersection, the decision for a crossing without a connection or a connection of all four edges is taken randomly. In Fig. 1b the connection nodes are marked with a C.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Data generation</figcaption></figure>

**Add Components.** Each node of the graph (which does not represent a intersection node) is assigned to a component. The random selection of components can be restricted by the number of edges of a node (e.g. heat exchanger are usually supplied with 4 connections).

**Derive Diagram from Graph.** The diagram is generated from the created graph. The selected symbols are drawn on a white background and the connection lines between the symbols are added, whereas each symbol size is randomly scaled (cf. Fig. 2).

<figure><img src="image_4.png" /><figcaption>Fig. 2. A generated diagram based on a 3 × 5 grid without noise</figcaption></figure>

## 3.2 Noise

Discussions with domain experts as well as the review of provided real HVAC diagrams show that the quality varies greatly and various interfering factors can occur. Digitized building plans can be available as scans, photographs or even as hand drawings. In addition to reduced resolution, scanned and photographed images often contain folded edges, slight noise or rotation, distortion or information added by hand. In order to simulate these issues in our synthetically generated diagrams and to maximize the robustness of the application, several disturbance factors were implemented as filter and can be added to the diagrams.
A random line filter, for example, represents the folded edges and inserted rect-
angles the grouping of strongly interacting system components. Slight rotations
of the input diagram, colored backgrounds and connection lines and general
noise like Gaussian or Jpeg compression are only a few more possible filters. An
example of a diagram with noise is shown in Fig. 3.

<figure><img src="image_5.png" /><figcaption>Fig. 3. A generated diagram based on a 3 × 5 grid with noise</figcaption></figure>

Besides the changeable diagram size and multiple intersection types, especially the variation of the symbol size within a diagram and the addable noise should increase the robustness of the object detection algorithm.

4 Developed Approach

4.1 Symbol Detection

**Template Matching.** A conventional and still common approach for detecting objects in images or videos is Template Matching (TM) [9, 10]. A given template image is compared with a section of the same size as the template in a larger input image and the areas most similar to the template are identified. TM is characterized by a simple algorithm and requires little computational power for simple applications. Initial tests and results can be implemented and evaluated within a few lines of code using Python libraries like OpenCV [3]. With simple geometric shapes (e.g. squares, circles), the TM provides good results but difficulties arise even with even small differences (e.g. aspect ratio) between the object and the template. The scaling or rotation of the object can also lead to a lack of correspondence between template and object and thus strongly influences the object recognition results.

A brief manual evaluation of the TM approach was conducted for the gen-
erated, as well for real world HVAC diagrams. In the case of the generated
diagrams with several superimposed noise filters and varying component sizes,
almost all of the components were recognized. In contrast, the TM approach for
components of real world diagrams has not delivered satisfying results. The very
few existing real diagrams demonstrate, that even for diagrams from the same
manufacturer there are small variations between similar components regarding the aspect ratio. Even an optimization approach to rotate the templates and vary the aspect ratio of the templates slightly did not lead to better results. Driven by the poor performance in real world applications the focus was switched on a Deep Learning approach using Faster R-CNN as described in the following.

**Faster R-CNN.** Deep Learning approaches are state of the art for many complex computer vision problems such as image segmentation, object detection and classification. This is possible due to the level of abstraction achieved by those models, which not only improves the robustness of a system, but also allows a great transferability between different domains of application. In order to locate and classify components within the HVAC diagram, we have selected a commonly used architecture of deep neural networks called Faster R-CNN [7]. In our context it is used to fulfill two tasks: perform object detection (create bounding boxes) and classification of objects (e.g. pump, valve, etc.). On an abstract level the network architecture is composed of three neural networks [4,7]:

1. Feature Network to generate representing features from the image

2. Region Proposal Network (RPN) to generate bounding boxes for region of interest (ROI)

3. Detection Network to inference object class for each ROI

The model is trained with data from our data generator (cf. Sect. 3) and the characteristics in Table 1.

<table><caption>**Table 1.** Characteristics of generated dataset and model training</caption><tr><td>Number of images (train)</td><td>1000</td></tr><tr><td>Number of images (test)</td><td>500</td></tr><tr><td>Symbol pool size</td><td>18 unique components (e.g. valve, damper)</td></tr><tr><td>Grid size for symbol placement</td><td>Height: 3-5, width: 3-5</td></tr><tr><td>Image resolution [px]</td><td>Height: 600-900, width: 600-900</td></tr><tr><td>Batch size</td><td>4</td></tr><tr><td>Base learning rate</td><td>0.001</td></tr><tr><td>Learning rate decay</td><td>0.05</td></tr><tr><td>Epochs</td><td>2000</td></tr></table>

For inference a single HVAC diagram is fed into the network, which returns the results in the Common Objects in Context (COCO) data format [6]. It contains the information of detected components as well as the corresponding bounding boxes in form of a defined JSON-format.
## 4.2 Line Detection

To create a robust line detection we decided to follow a multi-steps approach as described in the following.

**Preprocessing.** Real world diagrams can have all kinds of noise and artifacts. To improve the robustness of the line detection the first step is to slightly blur the image. In a second step we apply object detection to get the bounding boxes of all symbols. Subsequently the bounding boxes are replaced by a white rectangle with a black border. By replacing the original components we reduce the complexity of the image in terms of number of lines and receive a "skeleton" of the diagram. In the next step the image gets gray-scaled and binarized.

**Processing.** From the binarized image a base graph is created using a skeleton network [1] which subsequently gets corrected and improved by multiple postprocessing steps as described in the following.

**Postprocessing.** The base graph can contain several incorrect connections and is cleaned based on different rules. Some edges have the same start and ending nodes, which has no meaning in our context and thus they are removed. Sometimes multiple nodes are found very close or within the region of a detected symbol (bounding box). Adding a margin in all directions of the bounding box, these nodes are united into a single node in the center of the bounding box. The new node is named based on the symbol type and position to match the naming schema of the data generator. Subsequently all nodes that do not represent components are checked if they are either 3- or 4-way connections and the node names are updated accordingly to match the data generator naming schema. In a final step all two-way connections are removed from the graph, as they don't offer important information.

# 5 Results

The evaluation of results is performed on the final graph, which describes the connection of all detected components within the processed diagram. As a metric we use the *jaccard similarity* in order to describe how similar the created graph and the ground truth graph (derived from data generator) are. Given two sets, A and B, the *jaccard similarity* is defined as the size of the intersection of set A and set B (i.e. the number of common elements) over the size of the union of set A and set B (i.e. the number of unique elements). Using graphs, either the edges or the nodes can considered to be the elements, whereas each edge represents a connection between two components and nodes represent symbols. Two edges are considered "equal" if they share the same starting and endpoint. The *jaccard similarity* returns a value between 0% and 100% and can thus be easily interpreted.
The evaluation is performed with 500 test diagrams produced by the data generator. Each diagram is processed individually and the *jaccard similarity* is calculated. As it can be seen in Table 2, a distinction is made between the evaluation of nodes (symbols) and edges (connections) and whether the whole diagram is correctly recognized or not. Out of 500 diagrams, almost 98.5% were recognized completely correct, meaning no manual effort is needed for correction. Even in the case of incorrect detected diagrams, on average most symbols (83%) and connections (79%) are recognized and only little manual editing would be required for correction.

<table><caption>**Table 2.** Evaluation for 500 generated test diagrams</caption><thead><tr><td></td><td>Correct detected</td><td>Incorrect detected</td></tr></thead><tbody><tr><td>Total number</td><td>492 (98,4%)</td><td>8 (1,6%)</td></tr><tr><td>Jaccard similarity nodes [%]</td><td>100</td><td>82,75</td></tr><tr><td>Jaccard similarity edges [%]</td><td>100</td><td>79,38</td></tr></tbody></table>

For errors in the final diagram we identified two possible reasons for which a description and an example is given in the following. The data generator creates recursive connections (connecting a component to itself, cf. Fig. 4a), a scenario which we most probably won't face in real applications and therefore no further efforts were made to solve the problem. Another possible reason for errors can be found in the usage of the Faster R-CNN for symbol detection. All input images are resized to a defined resolution before the symbol detection is applied. For big images (high resolution) which contain small symbols (e.g. compressor), the

<figure><img src="image_6.png" /><figcaption>**Fig. 4.** Example of detected errors detected lines (graph edges) detected symbols (graph nodes) real connection lines (Color figure online)</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>**Fig. 5.** Results with real diagram detected lines (graph edges) detected symbols (graph nodes) real connection lines (Color figure online)</figcaption></figure>

resizing removes information to such an extent, that the model can't detect the symbol anymore. For the given example in Fig. 4b the original symbol for the compressor is hidden underneath multiple (incorrectly) detected graph nodes.

Additionally to the automated evaluation of generated data we manually inspected the results for a real diagram. Text elements were removed before processing, but no other manual steps were conducted. The final graph in Fig. 5 seems to represent the diagram almost completely correct. Reviewing the detected symbols we found only a single incorrect match: The heater was classified as a damper. Taking into consideration the probabilities, a clear trend can be observed: All correct symbols were detected with a probability over 95%, whereas the incorrectly classified damper reaches less than 75%.

# 6 Conclusion

The developed approach for symbol and line detection achieves very good results with diagrams created by the data generator and converts them into a machine-readable graph. Using graphs enables a flexible utilization for downstream applications. The achieved results with real world diagrams the developed approach are promising to significantly reduce the amount of work of implementing BIM and creating digital twins during a buildings lifecycle, especially in retrofit cases. Despite the promising results, the applicability for real diagrams has not yet been extensively tested. Especially in cases of real diagrams with a high resolution the symbol detection becomes a bottleneck in our developed approach. Although it is a first step towards an automated system, in future research additional aspects need to be put into consideration:

- Tackle detection problems of small symbols in large diagrams with a stepwise approach using a sliding window
- Improve the automation of the overall process by increasing the number of symbols and using data augmentation for the data generator

- Implement optical character recognition (OCR) to extract text information such as datapoint names (enables automatic datapoint mapping for building automation systems)

**Acknowledgements.** Supported by: BMWi (Federal Ministry for Economic Affairs and Energy), promotional reference 03ET1567A.

## References

1. Skeleton network. https://github.com/Image-Py/sknw

2. Angles, R., Gutierrez, C.: Survey of graph database models. ACM Comput. Surv. **40**(1), 1–39 (2008). https://doi.org/10.1145/1322432.1322433

3. Bradski, G.: The openCV library. Dr. Dobb's J. Softw. Tools Prof. Program. **25**(11), 120–123 (2000)

4. Chen, X., Gupta, A.: An implementation of faster RCNN with study for region sampling. arXiv preprint arXiv:1702.02138 (2017)

5. Gilani, S., Quinn, C., McArthur, J.J.: A review of ontologies for smart and continuous commissioning. arXiv preprint arXiv:2205.07636 (2022)

6. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, David, Pajdla, Tomas, Schiele, Bernt, Tuytelaars, Tinne (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1-48

7. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-CNN: towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. **28** (2015). https://doi.org/10.48550/ARXIV.1506.01497

8. Stinner, F., Wiecek, M., Baranski, M., Kümpel, A., Müller, D.: Automatic digital twin data model generation of building energy systems from piping and instrumentation diagrams. arXiv preprint arXiv:2108.13912 (2021)

9. Sun, Y., Mao, X., Hong, S., Xu, W., Gui, G.: Template matching-based method for intelligent invoice information identification. IEEE Access **7**, 28392–28401 (2019). https://doi.org/10.1109/ACCESS.2019.2901943

10. Thomas, L.S.V., Gehrig, J.: Multi-template matching: a versatile tool for object-localization in microscopy images. BMC Bioinform. **21**(1), 1–8 (2020). https://doi.org/10.1186/s12859-020-3363-7

11. Vanlande, R., Nicolle, C., Cruz, C.: IFC and building lifecycle management. Autom. Constr. **18**(1), 70–78 (2008). https://doi.org/10.1016/j.autcon.2008.05.001

12. Volk, R., Stengel, J., Schultmann, F.: Building information modeling (BIM) for existing buildings—literature review and future needs. Autom. Constr. **38**, 109–127 (2014). https://doi.org/10.1016/j.autcon.2013.10.023