SUGAMAN: describing floor plans for visually impaired by annotation learning and proximity‐based grammar

IET Image ProcessingVolume 13, Issue 13 p. 2623-2635 Research ArticleFree Access SUGAMAN: describing floor plans for visually impaired by annotation learning and proximity-based grammar Shreya Goyal, Shreya Goyal Department of Computer Science and Engineering, Indian Institute of Technology Jodhpur, IndiaSearch for more papers by this authorSatya Bhavsar, Satya Bhavsar Department of Computer Science and Engineering, Indian Institute of Technology Jodhpur, IndiaSearch for more papers by this authorShreya Patel, Shreya Patel Department of Computer Science and Engineering, Indian Institute of Technology Jodhpur, IndiaSearch for more papers by this authorChiranjoy Chattopadhyay, Corresponding Author Chiranjoy Chattopadhyay chiranjoy@iitj.ac.in orcid.org/0000-0002-3431-0483 Department of Computer Science and Engineering, Indian Institute of Technology Jodhpur, IndiaSearch for more papers by this authorGaurav Bhatnagar, Gaurav Bhatnagar Department of Mathematics, Indian Institute of Technology Jodhpur, IndiaSearch for more papers by this author Shreya Goyal, Shreya Goyal Department of Computer Science and Engineering, Indian Institute of Technology Jodhpur, IndiaSearch for more papers by this authorSatya Bhavsar, Satya Bhavsar Department of Computer Science and Engineering, Indian Institute of Technology Jodhpur, IndiaSearch for more papers by this authorShreya Patel, Shreya Patel Department of Computer Science and Engineering, Indian Institute of Technology Jodhpur, IndiaSearch for more papers by this authorChiranjoy Chattopadhyay, Corresponding Author Chiranjoy Chattopadhyay chiranjoy@iitj.ac.in orcid.org/0000-0002-3431-0483 Department of Computer Science and Engineering, Indian Institute of Technology Jodhpur, IndiaSearch for more papers by this authorGaurav Bhatnagar, Gaurav Bhatnagar Department of Mathematics, Indian Institute of Technology Jodhpur, IndiaSearch for more papers by this author First published: 21 October 2019 https://doi.org/10.1049/iet-ipr.2018.5627Citations: 8AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract In this study, the authors propose a framework SUGAMAN (Supervised and Unified framework using Grammar and Annotation Model for Access and Navigation). SUGAMAN is a Hindi word meaning ‘easy passage from one place to another’. SUGAMAN synthesises textual description from a given floor plan image, usable by visually impaired to navigate by understanding the arrangement of rooms and furniture. It is the first framework for describing a floor plan and giving direction for obstacle-free movement within a building. The model learns five classes of room categories from 1355 room image samples under a supervised learning paradigm. These learned annotations are fed into a description synthesis framework to yield a holistic description of a floor plan image. Authors demonstrate the performance of various supervised classifiers on room learning and provided a comparative analysis of system generated and human-written descriptions. The contribution of this study includes a novel framework for description generation from document images with graphics while proposing a new feature representing the floor plans, text annotations for a publicly available data set, and an algorithm for door to door obstacle avoidance navigation. This work can be applied to areas like understanding floor plans and design of historical monuments, and retrieval. 1 Introduction One of the primary goals of a combined framework involving computer vision (CV) and natural language processing (NLP) is to understand an image and describe it. The techniques available in CV and digital image processing helps to localise the objects in the image, identify key attributes and provide a relationship among them. On the other hand, NLP provides us with an end to end description of that image and thereby connecting the output of CV with the text. For the purpose of having a better understanding of a floor plan image, we propose Supervised and Unified framework using Grammar and Annotation Model for Access and Navigation (SUGAMAN), which is an attempt to connect these two modalities, i.e. CV and NLP, in the context of document images and text data. SUGAMAN is a Hindi word which translates to easy passage from one place to another. Apart from describing the general information about the floor plan images, it also generates room to room navigation information, while avoiding the obstacles. This navigation information can be very useful for visually impaired people as it becomes difficult for them to move in an indoor environment. It will be really helpful for them if there is a system that tells about the surrounding environment in natural language. SUGAMAN generates such a natural language description of an indoor environment from building floor plan images, which gives a detailed idea of the indoor environment. Here the input is a building floor plan image and the output is a textual description of the same. The description includes detail about the (i) rooms, (ii) connectivity among the rooms, (iii) type of decor within every room, and (iv) their relative position, and (v) navigational information while avoiding obstacles. In the past few years, unconventional documents like floor plans, engineering drawing gained a lot of attention from the document analysis and recognition (DAR) community. Engineering drawings contains many symbols, texts, line drawings, which need to be recognised and understood. Understanding of circuit diagrams, floor plan images, machine designs, building diagrams, maps etc. has its own importance. While researchers have looked into the problems of segmentation, symbol spotting, text recognition, symbol classification etc., the problem of narration synthesis from those documents was overlooked. Generating a textual description (narration) from an engineering drawing document image can be very useful for a layman to understand the document. As a particular example, building floor plan images involve a lot of technicalities such as symbols, dimension of the building, its design, orientation, as well as the interior decors, which are not known by common users. Various techniques have been proposed for its wall segmentation, text, room and decor segmentation in order to understand them. Even though text synthesis has been an area covered under NLP for years and also explored with real world images, document images were always ignored. Fig. 1 exemplifies the problem and the potential solution for a real-world floor plan images. It also depicts the navigation path to be generated and to be textually described. The key characteristics that makes this work unique are: (i) proposing a unified framework for narration synthesis from floor plan images, (ii) improvement in the previously available techniques for decor characterisation, (iii) proposal of a novel feature to represent a room within a floor plan, (iv) learning the room annotations for room classification, and (v) augmentation of an publicly available data set by annotating floor plan images with textual descriptions. Fig. 1Open in figure viewerPowerPoint Introducing the problem of narration synthesis from a given input floor plan image with the navigation path The paper is organised as follows: in Section 2 we discuss the related works published in the literature. Overview of the proposed system is given in Section 3. Description about the data set used is given in Section 4. Room annotation learning and classification framework is presented in Section 5. The method of description synthesis is discussed in Section 6. Experimental results are discussed in Sections 7 and 8, while comparative study with existing techniques is presented in Section 9. Finally, the paper is concluded in Section 10. 2 Related work In this section, existing works proposed in the literature, which are closely related to the proposed framework are presented. 2.1 Document image processing and recognition Symbol spotting, character recognition and feature extraction has been an evergreen research topic for classification of document images. An overview of graphics symbol recognition can be found in [1]. In this matter [2] provides a detailed state of the art survey for symbol spotting techniques and there performance analysis. In [3], a class of drawings are analyzed which includes flow charts by pre-processing scanned binary images, symbols and connection lines followed by extraction of features like shapes, orientation of lines and connections. Trier et al. [4] have also imparted knowledge based systems for interpreting symbols in engineering drawings. In [5], with respect to engineering drawings, an approach for detecting and classifying dashed line segments were proposed. In [6], a new rule based algorithm was proposed to differentiate between text and graphics part from engineering drawings. In a floor plan, the decors are an inseparable component and visually similar to characters. Properties of characters and their recognition methods could be adopted for recognising decors. The authors of [7] used topological analysis method to recognise characters and lines from gray scale scanned images of hand printed documents, whose information has been lost after binarisation. In [8], CCA and histogram based thresholding is applied to separate text and graphics and Hough transform is used to group together components into logical strings, which was later improved by Tombre et al. [9] through right choice of threshold and its stability. Language invariant script recognition was proposed in [10] using spatial relationships of features. Two classes of languages are determined using optical density distribution and most frequently occurring word shapes characteristics. Decors symbols can also be considered as line drawing images. In that context Freeman [11] discusses various forms of line drawing representations, processing of derived line drawing structures by extracting their features like chain coding scheme, polygon approximation. Qureshi et al. [12] have proposed a solution for symbol spotting using a graph representation of graphical documents. Dutta et al. [13] has proposed a symbol spotting technique in graphical documents, in which graph represents the document and a sub-graph matching is used to spot symbols. Viola and Jones [14] proposed a learning algorithm by selecting salient visual features and a novel representation called ‘integral image’, which allows the features used by detector to be computed very quickly. They combined increasingly more complex classifiers in a ‘cascade’, which allows background regions of the image to be quickly discarded speeding up object detection. Reference [15], proposed a new rotated Haar like features was proposed, which improves the accuracy of object detection by great extent. In [16], Joachim discussed about bag of words approach for objects categorisation. However in another work [17], an image is divided into blocks, and blocks with higher density are considered as text by grouping them, where key points are extracted by using FAST [18] method. In the context of floor plans text/graphic segmentation were performed in [19-21]. In [22], a new algorithm to segment the ancient maps and floor plans was proposed by removing non-textual elements and recognsizing characters to identify the plans. In [21], various rooms are detected and labelled by optical character recognition (OCR) on localised text regions. Heras et al. [23] have proposed a Statistical patch-based Bag of visual words (BoVWs) model to segment floor plan image. Attributed graph of line segments is generated with nodes labelled for wall segmentation, followed by Depth First Search (DFS) search to obtain walls in [23]. In [23], doors and windows are detected using symbol spotting techniques using SURF feature by detecting the key points in the image. In [24], walls in a floor plan image are detected by exploiting the properties of Hough transform on vectorised image. Also they have used BoVWs for the same task and later A* search is performed to find a path between doors. In [23], rooms are detected in the floor plan images by finding the closed regions in WDWC graph, preceded by removal of all the terminals of the graph. Wall contained image is decomposed into convex regions to detect the rooms and holes in the polygons are resolved. Later convex regions are checked for over segmentation by identifying the fitting rectangle of that region in [24]. In [25], a method for recognizing ‘uniform’ binary patterns was proposed. 2.2 Image description generation Description generation from images has been an interesting area for aligning images to their corresponding text. Although literature is available on natural images, description of document images is still an untouched domain. In one of our earlier work [26] we have introduced the problem of description generation from floor plan. An extensive survey of the existing techniques is given in [27]. Vinyals et al. [28] presented a generative model, based on deep recurrent architecture, which takes an input image I, and trained to maximise the likelihood of producing a target sequence of words . In the same line [29] discusses a model to generate descriptions from images, which automatically learns to describe the content of images. The work in [30] has presented a holistic data-driven approach to image description generation, which uses vast amount of image data and associated description available over internet. They recognise and predict the contents of an image and then use existing human composed description to generate natural captions for images. In [31, 32], authors have generated dense description of images, by developing a deep neural network model. They introduced a multimodal recurrent neural network architecture that takes an input image and generates its description in text. In [33], authors have used image meta-data for automatically generating textual image collection descriptions that include both image content and context information. Moreover, they convert and expand the metadata, by using publicly available information and services over internet. Kulkarni et al. [34] generated description of images by detecting objects, modifiers (adjectives) and spatial relationships (prepositions) in an image, comparing and smoothing these detection using pre-available description text. In the same context, Farhadi et al. [35] proposed a system that can obtain a score by linking an image to a sentence. In [30], authors have proposed a data-driven approach for description generation by giving a query image and retrieving existing human-composed phrases which describe similar images. Combining those phrases they generate a description for the given image. Verma and Jawahar [36] have proposed a system that achieves task like given an image, generating a textual description and vice versa, where both approaches are retrieval based. In the same line Zhu et al. [37] provide an approach to align visual contents of a movie release to their corresponding book by providing a description of the visuals. The image description is generated by creating visual dependency representation of natural images in [38]. Natural language description generation is also done for video for their retrieval purpose in [39], in which they capture relations between keywords associated with videos. Evaluation of machine translation with human-generated description is also necessary. For that purpose several metrics for example Bilingual Evaluation Understudy (BLEU) [40], Recall-Oriented Understudy for Gisting Evaluation (ROUGE) [41], Metric for Evaluation of Translation with Explicit Ordering (METEOR) [42] etc., have been proposed. In [43], authors provided a correlation between automatic metrics and human judgments, using previously mentioned metrics and their variants. Efforts have been made to connect two domains, NLP and document image analysis. For example in [44] authors have gained performance by integrating features from linguistic analysis, image text recognition and image layout analysis. Looking at the existing literature we see that even though description synthesis from natural images has become common, the same from document images is still not there, and thus we propose SUGAMAN to bridge the gap. SUGAMAN is an extension of our earlier work [26]. The key differences between SUGAMAN and [26] are: (i) a feature-based automatic room annotation learning method is proposed, as compared to OCR-based method, (ii) an improved proximity-based sentence model is proposed in SUGAMAN, instead of a template-based model, (iii) introduction of room to room navigational information for obstacle-free movement. 3 System overview Fig. 2 shows a block diagram depicting various modules and work-flow within SUGAMAN. The whole system is divided into two stages (i) room annotation learning and (ii) description synthesis. At first, room semantic information is extracted by room segmentation process, which gives all the required information about an input floor plan image. For example, individual room area, door information which gives room neighbourhood information, room locations (room coordinates). With those room locations, the floor plan image is partitioned into individual room image and taken as the sample for room annotations learning. Decor characterisation is applied over these room images and decors present in a room are labelled. We have proposed a local orientation and frequency descriptor (LOFD) feature, which is extracted from these room samples for automatic room annotation. A classifier is trained using LOFD feature matrix of room samples by assigning class labels to them. After this a new input image is taken as a test sample, features are extracted and room annotations are identified for it using previously trained model. An XML file is generated using the semantic information extracted by room segmentation and room classification. By parsing this XML file, textual description is generated. SUGAMAN also gives navigation path within the entire floor plan, starting from the entry door to the building. All such information about floor plan and navigation are fed to the proposed grammar model. The first stage of the proposed description synthesis method deals with ‘what to say’ about the floor plan and the second stage will deal with ‘how to say it’. For ease of understanding, in rest of the paper, we demonstrate all our analysis on the input image shown in Fig. 1. Later, in the experimental results, we also show the results on other floor plan images. Next we discuss the data set used for experimentation. Fig. 2Open in figure viewerPowerPoint Block diagram depicting various modules and work-flow within SUGAMAN 4 Floor plan data set In the literature, three floor-plan data sets were proposed, namely (i) Systems Evaluation SYnthetic Documents (SESYD) [45] (ii) Computer Vision Center Floor Plan (CVC-FP) [46] and (iii) Repository Of BuildIng plaNs (ROBIN) [47]. The SESYD has ten classes of floor plans, with 100 samples/class. On the other hand, CVC-FP has 122 scanned floor plan documents divided into four categories based on origin and style. In ROBIN there are three broad categories, which are different from each other in terms of the number and type of rooms present in a floor plan. The three categories are (i) three room, (iv) four room, and five room floor plans. Each category is further classified into ten sub-categories depending upon the global layout of the floor plan. ROBIN helps in better visualisation of the floor plans and aids in efficient capturing of various high-level features while fine-grained retrieval. Since ROBIN has significant number of floor plans, as well as intra-class similarity and inter-class dissimilarity, it is suitable in our case. However, in ROBIN there is no textual description available for a given floor plan. For our purpose we further augmented ROBIN (A-ROBIN) data set by introducing textual description for each floor plan image. 4.1 A-ROBIN In order to understand the floor plan images better and produce automated textual descriptionnarrating them we propose a data set A-ROBIN. In A-ROBIN there are fourhuman-written descriptions for each image in ROBIN data set. In literature,there are a few image data sets, e.g. Flicker8k [48], COCO data set [49],which have associated descriptions of the image samples. However, these datasets describe the natural images. In the context of document images, data setconsisting description of floor plan images was lacking. Hence we require anovel data set which contains annotations and descriptions for document images.The descriptions for a floor plan images were collected from the volunteers inthe following manner. Each volunteer was supplied with a set of ten images in aGoogle form and asked to describe them in their own words based on a set ofinstructions. Each form was given to four volunteers, so that each image hasfour set of descriptions. These descriptions focus on the rooms and their decorcontent in the floor plan images. Also they focus on their relative positioningin respective rooms and relative position of each room within the floor planimage. These descriptions vary in the sequence of the information given, thedetails of information provided, the sentence conjunctions and the vocabularyused for components in floor plan image. Fig. 3 shows one of the floor plan and its corresponding descriptions. Fig. 3Open in figure viewerPowerPoint Example of a sample floor plan image from ROBIN data set andthe annotation collected for the same to synthesiseA-ROBIN It can be observed that different descriptions of the same image vary inthe amount of information provided, the sequence of describing each room, thenames for each decor item could be different for different user. Also, a roommay have variations in its name for different user. The length of thedescriptions provided for each image is also varied. After the descriptions werecollected, each set of four descriptions was tagged to their respective imagesusing image identifiers. The volunteers were directed to annotate about the (i) entire plan (number of rooms, name of the rooms), (ii) individual rooms – name, decor contained, relative size (small, medium, large, etc.), (iii) relative positioning of decors (north, east, west, south, aligned with wall, adjacent with something, etc.), (iv) relative position of each rooms (north, east, west, south, adjacent with bathroom, north of kitchen, etc.). Apart from this information, the description of a volunteer may also provide additional information as per his/her choice. For 510 images in ROBIN data sets, there were 2040 descriptions collected in total. The data set is tokenised and pre-processed for further processing. In our experiments we have compared machine-generated description with these descriptions and an analysis regarding closeness of machine translations with human-written descriptions is done. Next, we describe the steps in our framework in detail. 5 Semantic segmentation and room classification In all the previous approaches available in the literature, rooms have been classified by recognizing the textual label present in the floor plan image using OCR techniques. Room classification in floor plans is not done by extracting salient feature from it. Room classification on the basis of their functionality is very useful in building information modelling. When a person enters a room in a house, he or she tells the functionality (class) of the room by looking at the decor items present inside the room. This inspired us to propose a unique feature for room classification. We have proposed a new feature called LOFD, which represents the frequency of decors present in a room and their normalised distance from the centre of the room. We proposed room classification approach as a five class classification problem, which annotates each room in a floor plan into one of the five classes namely, BEDROOM (label-1), BATHROOM (label-2), ENTRY (label-3), KITCHEN (label-4), HALL (label-5). The following subsections describe the details of room label learning and classification. 5.1 Room segmentation We have adopted the technique proposed in [50] for the identification of rooms. Walls are detected by performing morphological closing on the input floor plan image (see Fig. 4c). To delineate room boundaries, we detect doors using scale-invariant features and close the gaps in wall image corresponding to the door locations. To obtain the rooms, we identified the connected components in the wall image by applying flood fill technique. The obtained connected components are the required rooms and their locations are obtained. Also, we calculate the areas of the respective rooms (polygon area), converted them into square feet (taking 100 pixels = 1 ft) and store all the information obtained, that is neighbourhood, room area, room location coordinates, in a separate data structure. Fig. 4Open in figure viewerPowerPoint Room segmentation and room partitioning process 5.2 Floor plan partitioning A floor plan image is partitioned into rooms using the room coordinates extracted from the previous steps as shown in Fig. 4d. These individual room images are the samples taken for training the room annotations. We have applied decor characterisation in further stages on each of these individual room images to extract the features. 5.3 Decor classification In this section we describe the procedure employed for decor characterisation and their classification. Fig. 5 shows the 12 decor symbols used in the data set [47]. We have improved the technique of decor characterisation proposed in [50] by applying sequence of morphological operations. The technique in [50] uses a normalised area ratio of largest three components of a decor symbol for classification and characterisation of decors. We have improved the technique by first collecting ten different signatures for each symbol, taking a mean over them (symbols with different orientations) and stored them in a signature library. During classification, we first preprocess the symbol by applying a sequence of morphological operations (erosion and dilation), so that the symbol does not have broken lines. Then we applied blob detection over the image and cropped each decor symbol for signature comparison. Now we compare the test image's signature with the signature stored in library and closest one is classified in its respective category. This modification in the technique greatly improved the classification accuracy for some symbols. Fig. 4b depicts the detection of symbols in the floor plan input image Fig. 4a with bounding boxes. These decors are classified in their respective categories shown in Fig. 5. Fig. 5Open in figure viewerPowerPoint Twelve classes of Decor models used in the experiments 5.4 Local orientation and frequency descriptor Once the decors inside a room are recognised, we compute the feature to classify a room. For room classification, we proposed a new feature named LOFD. The LOFD feature is required since no other feature descriptor for example SIFT [51], LBP [25] or SURF [52] could capture the room images clearly. The LOFD feature is a vector containing the decor information of a room sample and their locally aggregated spatial information. Fig. 6 shows the LOFD feature matrix for the sample floor plan image. In LOFD, we have aggregated local information of room image in a vector form. LOFD is a compact representation of frequency of the decor items and normalised distance of their centres from centre of the room. The first 12 cells of the vectors are occupied by the 12 decor items from and next 12 cells are occupied by their normalised distances as where is the distance of each decor item from the centre of the room. Here, is calculated as (1)here is the normalised distance for each decor item, i is the count of each decor which may go up to k, which is the maximum number of a decor item in the room, dist is the Manhattan distance between room centre and decor centre , is the maximum of all the distances obtained for all the decors to normalise the distance value. Hence LOFD feature distinguishes each room uniquely by the frequency of each decor item and their spatial location in the room. Fig. 6Open in figure viewerPowerPoint Example of LOFD descriptor for four sample room images Since there are 12 decor models as shown in Fig. 5, first 12 elements of LOFD represents a count of one decor item. However, it is not necessary for any room to have all types of decor present, therefore LOFD is sparse in nature. In Fig. 6, depiction of the room image followed by the corresponding LOFD feature vector is shown. The coloured bar over each cell represents the frequency count for each decor item, while arrows in red represent their relative spatial location in the room. In the next section training of the classifier using LOFD feature for room classification is explained. 5.5 Room annotations learning and room classification Room annotations for training samples (divided the 1355 room images into 70 and 30% for training and testing, respectively) are learned by LOFD feature and classifier. For training purpose, we have manually annotated the room samples and used those annotations du