Tell2Design: A Dataset for Language-Guided Floor Plan Generation
Year: 2023

Abstract:
We consider the task of generating designs directly from natural language descriptions, and consider floor plan generation as the initial research area. Language conditional generative models have recently been very successful in generating high-quality artistic images. However, designs must satisfy different constraints that are not present in generating artistic images, particularly spatial and relational constraints. We make multiple contributions to initiate research on this task. First, we introduce a novel dataset, Tell2Design (T2D), which contains more than 80k floor plan designs associated with natural language instructions. Second, we propose a Sequence-toSequence model that can serve as a strong baseline for future research. Third, we benchmark this task with several text-conditional image generation models. We conclude by conducting human evaluations on the generated samples and providing an analysis of human performance. We hope our contributions will propel the research on language-guided design generation forward1.

heading: 1 Introduction
text: Recently, text-conditional generative AI models (Nichol et al., 2022; Saharia et al., 2022b; Ramesh et al., 2022; Dhariwal and Nichol, 2021; Ho et al., 2022) have demonstrated impressive results in generating high-fidelity images. Such models generally focus on understanding high-level visual concepts from sentence-level descriptions, and the generated images are valued for looking realistic and being creative, thereby being more suitable for generating artwork. However, besides less constrained generation like artworks, generating designs that meet various requirements specified
∗ Equal contribution † Most work done at NUS 1Code and dataset are available at https://github.com/
LengSicong/Tell2Design.
in natural languages is also much needed in practice (Stiny, 1980; Seneviratne et al., 2022; Zhang et al., 2022; Wei et al., 2022). In particular, a design process always involves interaction between users/clients, who define objectives, constraints, and requirements that should be met, and designers, who need to develop various solutions with domain-specific experiences and knowledge. For example, users may dictate their house design requirements in text and expect expert architects to perform the floor plan generation.
Previous research in layout generation aims to automate the process of layout design in different domains such as scientific documents, mobile UIs, indoor scenes, etc (Zhong et al., 2019; Deka et al., 2017; Song et al., 2015; Janoch et al., 2013; Xiao et al., 2013; Silberman et al., 2012; Cao et al., 2022; Wu et al., 2019). Most of them perform the generation either based on several hand-crafted constraints or by using unconstrained generation. In practice, it can be more convenient for users to indicate their preferences in natural language.
Among various design tasks, floor plan2 design, as shown in Figure 1 is of moderate complexity. However, it still intrinsically involves multiple rounds of communications between clients and de-
2Architectural floor plans, i.e., interior building layouts, are documents that indicate room types, room connections, room sizes, etc. They play a crucial role while designing, understanding, or remodeling indoor spaces (Liu et al., 2017).
14680
signers for specifying requirements, and requires a high level of precision and alignment to detail. AI systems that can learn to generate practically useful floor plan designs directly from natural languages will go a long way in reducing the protracted design process and making Generative AI directly usable for design by the end users.
To allow people without expertise to participate and further enhance the design process, we aim to enable users to design by “telling” instructions, with a specific focus on the floor plan domain as the initial area of research. This sets forth a new machine learning task where the model learns to generate floor plan designs directly from language instructions. However, this task brings up two technical challenges. First, a floor plan is a structured layout that needs three intrinsic components to be valid: (1) Semantics, which describes the functionality of rooms (e.g., for living or bathing); (2) Geometry, which indicates the shape and dimension of individual rooms; (3) Topology, which defines the connectivity among different rooms (Pizarro et al., 2022). Second, these instructions are expressed in natural languages, which, besides the diversity of expressions, inherently suffer from ambiguity, misleading information, and missing descriptions for intrinsic components.
To address the above challenges, we make multiple contributions to initiate research on the task of language-guided floor plan generation. First, we contribute a novel dataset, Tell2Design (T2D), to the research community. The T2D dataset contains more than 80k real floor plans from residential buildings. Each floor plan is associated with a set of language instructions that describes the intrinsic components of every room in a plan. An example from the dataset is illustrated in Figure 1. Second, we propose a Sequence-to-Sequence (Seq2Seq) approach as a solution to this task which also serves as a strong baseline for future research. Our approach is strengthened by a new strategy to explicitly incorporate the floor plan boundary constraint by transforming the outline into a box sequence. Third, in order to benchmark this novel task and evaluate our proposed approach, we implement strong baselines in text-conditional image generation on our T2D dataset and ask humans to perform the same task. The generation alignment with language instructions is evaluated both quantitatively and qualitatively. Finally, we discuss several future directions that are worth exploring based on our
experimental results. In summary, our main contributions are:
• We introduce a novel language-guided floor plan generation task along with the T2D dataset consisting of both natural humanannotated and large-scale artificially generated language instructions (Section 3).
• We propose a new approach that formulates the floor plan generation task as a Seq2Seq problem (Section 4).
• We provide adequate quantitative evaluations on all baselines and qualitative analysis of human evaluations and performances (Section 5).

heading: 2 Related Work
text: Text-Conditioned Image Generation Image generation is a well-studied problem, and the most popular techniques have been applied for both unconditional image generation and text-conditional settings. Early works apply auto-regressive models (Mansimov et al., 2015), or train GANs (Xu et al., 2018; Zhu et al., 2019; Tao et al., 2022; Zhang et al., 2021; Ye et al., 2021) with publicly available image captioning datasets to synthesize realistic images conditioned on sentence-level captions. Other works have adopted the VQ-VAE technique (Van Den Oord et al., 2017) to text-conditioned image generation by concatenating sequences of text tokens with image tokens and feeding them into autoregressive transformers (Ramesh et al., 2021; Ding et al., 2021; Aghajanyan et al., 2022). More recently, some works have applied diffusion models (Ho et al., 2020; Nichol and Dhariwal, 2021; Saharia et al., 2022c; Dhariwal and Nichol, 2021; Ho et al., 2022; Saharia et al., 2022a; Rombach et al., 2022; Nichol et al., 2022; Saharia et al., 2022b; Ramesh et al., 2022) and received wide success in image generation, outperforming other approaches in fidelity and diversity, without training instability and mode collapse issues (Brock et al., 2018; Dhariwal and Nichol, 2021; Ho et al., 2022). However, these models operate on extracting high-level visual concepts from the short text and produce artwork-like images that are expected to be realistic and creative, thereby not suitable for generating designs that must satisfy various user/client requirements.
Layout Generation Layout generation is essentially a design process that requires meeting domain-specific constraints, where the desirable
layouts could be documents, natural scenes, mobile phone UIs, and indoor scenes. For example, PubLayNet (Zhong et al., 2019) is proposed to generate machine-annotated scientific documents with five different element categories text, title, figure, list, and table. RICO (Deka et al., 2017) is introduced to develop user interface designs for mobile applications, which contains button, toolbar, etc. SUN RGB-D (Song et al., 2015) presents a combined scene-understanding task, including indoor scenes from three other datasets (Janoch et al., 2013; Xiao et al., 2013; Silberman et al., 2012). Moreover, ICVT (Cao et al., 2022) aims to produce advertisement poster layouts automatically, where the image background is given as input. The above methods are designed for different layout domains and cannot be directly applied to floor plan design. Moreover, none of them has considered generating the layout design directly from languages.
Floor Plan Generation Several methods have been proposed to generate floor plan designs automatically (Wu et al., 2018; Liu et al., 2013; Merrell et al., 2010; Hua, 2016; Wu et al., 2019; Chen et al., 2020; Chaillou, 2020). Most of these methods generate floor plans conditioned on certain constraints, such as room types, adjacencies, and boundaries. For example, Merrell et al. (2010) generate buildings with interior floor plans for computer graphics applications using Bayesian networks without considering any human preferences. Liu et al. (2013) present an interactive tool to generate desired floor plan following a set of manually defined rules. Hua (2016) particularly focus on generating floor plans with irregular regions. Wu et al. (2018) cast the generation as a mixed integer quadratic programming problem where some floor plan components are formulated into a set of inequality constraints. More recently, Wu et al. (2019) propose a CNN-based method to determine the location of different rooms given boundary images as a constraint. Chen et al. (2020) provide a small amount of template-based artificial verbal commands and manually parses them to scene graphs for guiding the generation.
In summary, existing methods represent the intrinsic components of floor plans in several specific formats as generation constraints. Some formats are straightforward, such as boundary images, but they only specify limited constraints and lead to less controllable generation. Other formats, such as scene graphs and inequalities, can incorporate more information but require specific domain knowledge
and extra-human efforts in pre-processing. We instead provide a unified and natural way of conditioning the floor plan generation with a set of language instructions that is much more flexible and user-friendly to characterize floor plans with various constraints.

heading: 3 Tell2Design Dataset
text: In this section, we introduce how we construct our T2D dataset, followed by the data analysis and a discussion of the main dataset challenges.

heading: 3.1 Task Definition
text: Given a set of language instructions describing a floor plan’s intrinsic components, our aim is to generate reasonable 2D floor plan designs that comply with the provided instructions.
Input & Output For each data sample, the input is a set of natural language instructions that characterize the key components of the corresponding floor plan design, which include: (1) Semantics specifies the type and functionality of each room. For example, a room as Kitchen is for cooking. (2) Geometry specifies the shape and dimension of each room. For residential buildings, it involves the room’s general orientation (e.g., the north, south, northeast, southwest), area in square feet, aspect ratio, etc. (3) Topology describes the relationships among different rooms. It can be divided into three categories: relative location, connectivity, and inclusion3. The desirable output is a structured interior layout that aligns with the input language instructions.

heading: 3.2 Floor Plan Collection
text: We use floor plans from RPLAN4 (Wu et al., 2019) to construct our Tell2Design dataset. We remove floor plans with rarely-appeared rooms and merge similar room types such as Second Room and Guest Room. As a result, 8 different room types (i.e., common room, bathroom, balcony, living room, master room, kitchen, storage, and dining room) and 80, 788 floor plans are selected for collecting language instructions. Each floor plan is converted into a 256×256 image where different pixel values
3As a result, language instructions specifying the above features for a floor plan lead to a document-level description in natural language. We compare the T2D dataset with several document-level NLP datasets in Appendix B.
4http://staff.ustc.edu.cn/~fuxm/projects/ DeepLayout/index.html
indicate different room types, from which we extract room-type labels and bounding boxes of each room to construct our dataset.

heading: 3.3 Language Instruction Collection
text: Human Instructions To collect real human language instructions, we hire crowdworkers from Amazon Mechanical Turk (MTurk)5 and ask them to write a set of instructions for each room according to a given floor plan image. The requested instructions should reflect the Semantic, Geometric, and Topological information of the floor plan, such that designers could ideally reproduce the floor plan layout according to the instructions. In particular, turkers are encouraged to include (but are not limited to) attributes such as room types, locations, sides, and relationships in their instructions. The definitions of these attributes are given as follows: The room type (e.g., bathroom and kitchen) specifies the functionality of a room. The room location specifies the global location of a room in the floor plan and can be described by phrases such as “north side” and “southeastern corner”. The room sides specify the length and width of a room (e.g., “8 feet wide and 10 feet long”). The room relationships specify the relative position of a room with other rooms such as “next to”, “between”, and “opposite” 6.
Due to the noisy nature of crowdsourcing annotations, we discard some low-quality annotations to ensure the overall quality of our datasets. To this end, we manually review each annotation and discard human instructions with: (1) incoherence, grammatical errors; (2) insufficient attributes; or (3) irrelevance to the given floor plan. As a result, we collect human instructions for 8, 220 floor plans, and 5, 051 of them are finally accepted after
5https://www.mturk.com/ 6To mimic the real-world scenarios, we do not provide any bounding box information and ask crowdworkers to make rough estimations of the room size from the given floor plan image only. We also do not restrict the format of text descriptions or require all the above attributes to be mentioned, leading to unstructured and diverse instructions.
Restricted
manual assessment to construct our dataset7.
Artificial Instructions In addition to the humanwritten instructions, we also generate language instructions artificially for the remaining 75, 737 floor plans from pre-defined templates. To ensure that the artificial instructions are as informative as human-written ones and include all the required components, we ask 5 educated volunteers with natural language processing (NLP) backgrounds to write language instructions for each room that appeared in the given floor plan. We then summarize their instructions into multiple templates and ask expert architectural designers for proofreading. Hence, each instruction template is ensured to be informative, grammatically correct, and coherent.
In summary, our T2D dataset consists of 5, 051 human-annotated and 75, 737 artificially-generated language instructions8.

heading: 3.4 Data Analysis
text: In this section, we analyze various aspects of Tell2Design to provide a more comprehensive understanding of the dataset.
Language Instructions Table 1 shows the statistics of the language instructions in our dataset. For each floor plan, the human instructions are organized in nearly 11 sentences consisting of 200 words on average. This includes around 30 words used to describe each room in more than 2 sentences. The artificially generated instructions follow a similar pattern with slightly more words.
To show the connections and differences between human and artificial instructions, we com-
7Our human instruction collection involves 5, 109 different workers with 1, 723 working hours, and each worker receives full compensation in alignment with the established standards of MTurk.
8More details on the language instruction collection can be found in Appendix C.
pare them for the same room type, Balcony, in Figure 2. Artificial instructions always exhibit complete information, including all three key components of a floor plan. They are also formatted in a structured expression, such as “on the ** side”, “** sqft with an aspect ratio of **”, and “next to”. However, human instructions are more diverse in expression but suffer from ambiguity and missing components.
Dataset Comparison In order to see our T2D dataset in perspective, we note its main differences with respect to other related datasets used for similar generation tasks9. T2D differs from other datasets in several perspectives: (1) T2D is the first large-scale dataset that aims to generate designs (i.e., floor plans) from direct user input natural language; (2) T2D has much longer text annotations (i.e., 256 words per instance) compared with other text-conditional generation datasets; (3) All text in T2D is written by humans or generated artificially, instead of being crawled from the internet.

heading: 3.5 Dataset Challenges
text: In this section, we discuss three main challenges of our collected T2D dataset. We hope this dataset can facilitate the research on both design generation and language understanding.
Design Generation under Constraints The first challenge is to perform the design generation under much stricter constraints compared with artworklike text-conditional image generation. Most works in text-conditional image generation operate on generating realistic and creative images that align with the main visual concepts represented by the short input text. However, creating a design from languages has much stricter requirements on precision and alignment to text details. In particular, the generated floor plan design should comply with constraints such as room type, location, size, and relationships, which are specified by users using natural languages. Our main results in Section 5 comparing different baselines demonstrate that existing text-conditional image generation techniques fail to follow detailed user requirements on this design task.
Fuzzy & Entangled Information The second challenge is to understand the big picture of the entire floor plan from document-level unstructured text with fuzzy and entangled information. Be-
9We provide detailed comparisons with tables in Appendix B.
sides the general abilities required for language understanding, such as entity recognition, coreference resolution, relation extraction, etc, models also need to collaborate with fuzzy individual room attributes and reason over entangled relationships among different rooms to understand the entire floor plan. Specifically, one language instruction usually either specifies fuzzy descriptions for a room’s Semantic and Geometric information such as “on the north side” and “at the southeast corner”, or indicates the relationship of one specific room with others like “next to” and “between”. The provided information in such instructions is coarse and relative, rather than complete and precise information like numerical coordinates. As a result, to determine the location of all rooms and design a reasonable floor plan, models must collaborate with fuzzy and entangled information residing in multiple instructions, and incorporate the boundary information. Human evaluations in Section 5.4 demonstrate that room relationships described in language instructions are the most challenging component to be understood and aligned with.
Noisy Human Instructions The third challenge comes from the ambiguous, incomplete, or misleading information in human instructions. As introduced in Section 3.3, the artificial instructions are template-based so that they always contain precise and coherent information. However, for humanwritten language instructions, ambiguous or noisy information always exists. For example, during human instruction collection, workers are asked to write natural sentences estimating some numericrelated attributes like room size and aspect ratio referring to the floor plan image, which may sometimes be inaccurate. Moreover, as previously discussed in Figure 2, other than the expression diversity, human instructions also exhibit ambiguous phrasing and incomplete information. It is thus more challenging for models to retrieve accurate, complete, and consistent information from human instructions.

heading: 4 T2D Model
text: In this section, we propose a simple yet effective method for language-guided floor plan generation. Unlike existing floor plan generation methods (Wu et al., 2019; Chen et al., 2020) that use a regression head to generate the bounding box of each room one at a time, we cast the floor plan genera-
tion task as a Seq2Seq problem under the encoderdecoder framework, where room bounding boxes are re-constructed into a structured target sequence. This way, our method can easily deal with various lengths of instructions for floor plans with different numbers of rooms.

heading: 4.1 Target Sequence Construction
text: Recall that our aim is to generate a floor plan layout from language instructions, where each room can be represented by a room-type label (e.g., bathroom and kitchen) and a bounding box. One bounding box can be determined by four values (x, y, h, w), which indicate the x and y coordinate of the center point, height (h), and width (w), respectively. To solve language-guided floor plan generation as a Seq2Seq problem, we treat the instructions as the input sequence and consider bounding boxes of rooms as the target sequence. Specifically, each of the continuous values (x, y, h, w) is discretized into integers between [0, 255], and the room type is given by the plain text in natural language, so that they can be naturally represented as a sequence of tokens. The target sequence is then constructed by grouping the room type and the bounding box together with certain special tokens. For example, the target sequence for a Balcony with the bounding box (87, 66, 18, 23) is given as follows:
[ Balcony | x coordinate = 87 | y coordi
nate = 66 | height = 18 | width = 23 ],
where the special tokens “[” and “]” are used to indicate the start and end of the target sequence for one room and “|” is used to separate different target components. We have also added semantic prefixes such as “x coordinate =” and “height =” before the values of bounding boxes to assist the target sequence generation. Finally, we concatenate the target sequences of all the rooms in a floor plan and add an <eos> token at the end to indicate the end of the overall target sequence.

heading: 4.2 Boundary Information Incorporation
text: The outline/boundary of a floor plan is one of the most important constraints in floor plan generation, which directly affects where each room should be placed and how different rooms should be aligned with the floor plan boundary. However, it is nontrivial to incorporate such boundary information into floor plan generation. Previous methods either fail to take the floor plan outline into account (Wu
et al., 2018; Liu et al., 2013; Merrell et al., 2010; Hua, 2016; Chen et al., 2020) or only consider the boundary image, ignoring all other constraints (Wu et al., 2019; Chaillou, 2020), leading to less controllable floor plan design.
In this work, we propose a novel approach to incorporate boundary information by representing the irregular outline as a set of boxes. The idea is to encode the boundary information by an enclosing box that is the minimum bounding region containing the entire floor plan and several exterior boxes that are inside the enclosing box but excluded from the floor plan. Figure 3 illustrate how the floor plan boundary can be characterized by the enclosing (in red) and exterior boxes (in yellow). This way, we have an enclosing box represented by (xen, yen, hen, wen) and M exterior boxes by (xexi , y ex i , h ex i , w ex i ). Then we adopt a similar strategy in Section 4.1 to represent the enclosing and exterior boxes in a sequence as follows:
+ xen yen hen wen - xex1 y ex 1 hex1 w ex 1 ... - x ex M y ex M h ex M w ex M ,
where the coordinates of the enclosing and exterior box are following the tokens “+” and “-”, respectively.
Finally, the above sequence is added after the input language instructions for training. Our experimental results in Section 5 show that the proposed boundary information incorporation strategy is effective in enhancing our Seq2Seq method to generate valid rooms that align well with the floor plan boundary.

heading: 4.3 Architecture, Objective and Inference
text: Treating the target sequences that we construct from floor plans as a text sequence, we turn to recent architectures and objective functions that have been effective in Seq2Seq language modeling.
Architecture We use the popular Transformerbased (Vaswani et al., 2017) encoder-decoder structure to build our Seq2Seq model for floor plan generation. The model is initialized by a pre-trained language model T5 (Raffel et al., 2020) for better language understanding abilities10.
Objective Similar to language modeling, our T2D model is trained to predict the next token, given an input sequence and preceding tokens, with a maximum likelihood objective function, i.e.,
max θ
L∑
j=1
logPθ (ỹj | x,y1:j−1) , (1)
where x is a set of instructions in natural language concatenated with the previously defined boundary sequence, y is the target bounding box sequence, and L is the target sequence length.
Inference At inference time, we sample11 tokens one by one from the model likelihood, i.e., P (ỹj | x,y1:j−1). The sequence generation ends once the <eos> token is sampled, and it is straightforward to parse the target sequence into predicted floor plans.

heading: 5 Experiments

heading: 5.1 Baselines
text: Since our T2D dataset is the first to consider language-guided floor plan generation, existing layout generation methods are not applicable to this task. To further illustrate the challenge of the design generation task and the difference with the existing text-conditional image generation problem, we adapt several state-of-the-art text-conditional image generation methods as baselines for comparison. In particular, we compare our method with the following:
• Obj-GAN (Li et al., 2019) is an object-driven attentive generative adversarial network that follows a two-step generation process. We apply the first-step box generator module, which takes the language as input and generates target objects’ bounding boxes (with class labels).
10We have also tried other pre-trained language models like Bart (Lewis et al., 2020), and preliminary experiments indicate that initializing our model with T5 leads to better performances.
11There are several common sampling strategies like Greedy Search, Beam Search, and Nucleus Sampling (Holtzman et al., 2019). We apply Greedy Search since it leads to better generation quality in our preliminary experiments.
• CogView (Ding et al., 2021) applies pre-trained VQ-VAE to transform the target image into a sequence of image tokens. Then the text and image tokens are concatenated together and fed to a Transformer decoder (i.e., GPT (Brown et al., 2020; Radford et al., 2019)) to generate text-conditional images.
• Imagen (Saharia et al., 2022b) is one of the state-of-the-art text-to-image generation models that build upon both large language models (e.g., T5) for text understanding and diffusion models for high-fidelity image generation.

heading: 5.2 Experimental Settings
text: Model Training For model training, we consider a Warm-up + Fine-tuning pipeline (Goyal et al., 2017), where the model is first warmed up on 75, 737 artificial instructions, and then fine-tuned on 2, 743 human instructions. To evaluate how floor plan generation methods generalize to unseen instructions, we use the remaining 2, 308 human instructions as the test set, such that there is no overlapping between annotators of the training set and the test set12.
Evaluation Metrics For testing, we use macro and micro Intersection over Union (IoU) scores between the ground-truth (GT) and generated floor plans at pixel level as the evaluation metrics, whose definitions are given as follows:
Micro IoU = ∑R
r=1 Ir∑R r=1 Ur ,Macro IoU = 1 R
R∑
r=1
Ir Ur ,
where the Ir and Ur, respectively, denote the intersection and union of the ground-truth and predicted rooms labeled as the r-th room type in a floor plan. R is the total number of room types. Macro IoU calculates the average IoU over different types of rooms, and Micro IoU calculates the global IoU by aggregating all rooms.
Since Obj-GAN and our T2D model generate bounding boxes rather than images, we use a simple strategy to transform the outputs of Obj-GAN and the T2D model into images without any further refinement for a fair comparison. Specifically, we paint each room in descending order in terms of the total area of the room type13 and different colors
12We provide more implementation details in Appendix A. 13Total area of the room type is computed by adding up the specific room type area across all floor plans in our dataset. This gives us the following order: living room, common room,
refer to different room types. Previous colors will be replaced by the subsequent paintings if there is an overlapping between bounding boxes (rooms). For image-based approaches (e.g., CogView and Imagen), we compute the maximized IoU scores by shifting the floor plan central point in the generated image.

heading: 5.3 Main Results
text: Table 2 shows the floor plan generation results on the T2D dataset, where T2D (w/o bd) indicates the T2D model without incorporating boundary information14. The T2D model achieves the highest IoU scores with a micro IoU of 54.34 and a macro IoU of 53.30, outperforming other baselines by a large margin. These can be attributed to our Seq2Seq model in controlling the target box sequence generation based on salient information extracted from language instructions. In contrast, text-conditional image generation methods fail to perform well. This is probably because those models are designed to generate artwork-like images with high-level visual concepts from the short text, instead of following multiple instructions with various constraints for a specific design.
master room, balcony, bathroom, kitchen, storage, dining room.
14We provide baseline generation samples in Appendix D.
When training only on artificial instructions while testing on human-written ones, our method cannot perform well. This indicates there is a language distribution gap between artificial and human instructions. Nevertheless, when artificial instructions are used for warming up before training on human instructions, the performance of our method is significantly improved with over 10 IoU scores increment. This suggests that despite the language gap, artificial and human instructions are mutually beneficial data portions during training.
In addition, in all the training settings, representing the floor plan boundary as a sequence of boxes consistently improves the performance of our Seq2Seq approach. This demonstrates that this strategy could be one of the possible solutions to incorporate the floor plan boundary.

heading: 5.4 Result Analysis
text: It is worth noting that the quantitative results indirectly evaluate how well the generated floor plans align with the language instructions since IoU scores essentially measure the overlap between generated and ground-truth floor plan layouts. Due to the complexity of our task, it is possible for the same language instruction to map to multiple floor plan designs. Therefore, a low IoU score does not necessarily mean a bad generation.
Human Evaluations To directly evaluate the alignment between generated floor plans and language instructions, we conduct human evaluations on a subset of the T2D test set, which consists of 100 randomly sampled instructions written by different annotators. For this purpose, we invite 5 volunteers with NLP backgrounds to evaluate the degree of alignment between source language instructions and target floor plans.
Specifically, we consider four partial alignment criteria in terms of room types, locations, sizes, and relationships. Each volunteer is asked to provide four ratings on a scale of 1 to 5, according to the
above-mentioned alignment criteria, respectively15. Besides, we also consider global alignment and ask our volunteers to justify whether the generated floor plan meets all the specifications in the instructions. We perform the above subjective evaluations for both T2D-generated and ground-truth floor plan designs.
Table 3 shows the human evaluation results. As can be seen, ground-truth floor plans get high ratings for all the partial alignment criteria, and 85% of them meet all the requirements specified in the instructions. This indicates our dataset contains high-quality human instructions that align well with the ground-truth floor plan designs. On the other hand, our T2D model receives no rating less than 3.5, indicating that at least 50% rooms with respect to their locations, sizes, and relationships can be correctly predicted. However, our method still has a gap with ground truth designs, especially in room location and relationships, which indicates the potential for improvements.
Human Performance To study human performance on the T2D task, we further ask our volunteers to design floor plans on 100 instances of the same subset used for human evaluations16. Table 4 reports the IoU scores for our T2D model and human performance. Humans generally achieve better IoU scores. However, even if human-generated floor plans intrinsically have much better alignment with the instructions, they only obtain around 63% IoU scores with the ground truths. This exposes the nature of design diversity, i.e., a set of language instructions can map to multiple plausible floor plan designs. Figure 4 provides a real example that both our method and humans follow the same instructions17 but generate different floor plans.
15For example, a rating of r with respect to room locations indicates that (r − 1)× 20% to r × 20% rooms in the floor plan follow their location specifications in the instructions.
16Specifically, volunteers are asked to draw a bounding box for each room, given the floor plan outline and the language instructions.
17The language instructions for this sample are shown in Figure 10 in Appendix D.
Restricted

heading: 6 Future Research
text: In the future, the following directions may be worth exploring to promote the performance or extend our task: (1) How to build robust language understanding models that can adapt to the presence of noise in human instructions or even locate and refine potentially inconsistent information? (2) How to explicitly incorporate the nature of design diversity and develop techniques for diverse floor plan design? (3) How to extend the language-guided floor plan generation task to more domains or more practical but challenging scenarios, where designs should be refined according to feedback from users/clients?

heading: 7 Conclusion
text: In this paper, we initiate the research of a novel language-guided design generation task, with a specific focus on the floor plan domain as a start. We formulate it as language-guided floor plan generation and introduce Tell2Design (T2D), a large-scale dataset that features floor plans with natural language instructions in describing user preferences. We propose a Seq2Seq model as a strong baseline and compare it with several text-conditional image generation models. Experimental results demonstrate that the design generation task brings up several challenges and is not well-solved by existing text-conditional image generation techniques. Human evaluations assessing the degree of alignment between text and design, along with the human performance on the task, expose the challenge of understanding fuzzy and entangled information, and the nature of design diversity in our task. We hope this paper will serve as a foundation and propel future research on the task of the language-guided design generation.
Limitations
The proposed T2D dataset has several limitations, which could be addressed in future work. First, it only considers and collects language instructions for the floor plan domain. Future work could extend this language-guided design generation task to other design domains such as documents, mobile UIs, etc. Second, it is limited in the scope of languages where we only collect instructions written in English. Future work could assess the generalizability of the T2D dataset to other languages. Third, although generating floor plan designs from languages exhibit diversity, we do not consider improving generation diversity at this moment. Future works could consider building frameworks that specifically aim at design diversity.
Ethics Statement
In this section, we discuss the main ethical considerations of Tell2Design (T2D): (1) Intellectual property protection. The floor plans of the T2D dataset are from the RPLAN (Wu et al., 2019) dataset. Our dataset should be only used for research purposes. (2) Privacy. The floor plan data sources are publicly available datasets, where private data from users and floor plans have been removed. Language instructions are either generated artificially or collected from Amazon Mechanical Turk, a legitimate crowd-sourcing service, and do not contain any personal information. (3) Compensation. During the language instruction collection, the salary for annotating each floor plan is determined by the instruction quality and Mturk labor compensation standard.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers, our meta-reviewer, and senior area chairs for their constructive comments and support of our work. We also gratefully acknowledge the support of NVIDIA AI Technology Center (NVAITC) for our research. This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Program (AISG Award No: AISG2-RP-2020-016).

heading: B Dataset Analysis
text: Floor Plan Statistics As shown in Tabel 5 and Figure 5, we present the statistics on the occurrence of each room type and the number of rooms per floor plan. There are 8 types of rooms in total. More than 92% of floor plans include at least 6 distinct rooms, and the most frequent room types are Common Room, Bathroom, Balcony, Living Room, Master Room, and Kitchen.
Dataset Comparison As shown in Table 8, compared with other-related layout generation datasets, our T2D dataset is the first to have language annotations and aims to generate layout designs directly
19https://github.com/jamesli1618/Obj-GAN; https: //github.com/THUDM/CogView
20https://github.com/lucidrains/imagen-pytorch
Restricted
Restricted
from languages. Since generating floor plan designs from language instructions can be naturally formulated into a text-conditional image generation problem, we compare our dataset with two benchmarking text-conditional image generation datasets in Table 6. We observe that our dataset is with a similar number of images with MSCOCO and Flickr30K but contains far longer text annotation for each image (i.e.T2D has 256 words on average describing each floor plan image). Moreover, as a set of language instructions for a floor plan results in a document-level text description, we compare our dataset with other document-level NLP datasets in Table 7. We hope that our dataset can also propel the research on document-level language understanding. It is shown that our dataset has a comparable total number of samples and words with the largest DocRED(Yao et al., 2019). More importantly, our “documents" are either humanannotated or artificially generated, instead of being crawled from the internet.

heading: C Dataset Collection Details
text: Human instruction We employ Amazon Mechanical Turk (MTurk)21 to let annotators write language instructions for a given RGB 2D floor plan. Amazon considers this web service "artificial intelligence," and it is applied in various fields, including data annotation, survey participation, content moderation, and more. The global workforce (called "turkers" in the lingo) is invited for a small reward to work on "Human Intelligence Tasks" (HITs), created from an XML description of the task from business companies or individual sponsors (called "requesters"). HITs can display a wide variety of content (e.g., text and images) and provide many APIs, e.g., buttons, checkboxes, and input fields for free text. In our case, turkers are required to fill the blank input fields in HITs with language instructions for each room, following our guidelines. A screenshot of one of our HITs is displayed in Figure 6. We also show a full example of human instructions in Figure 7.
Artificial Instruction The artificial instructions in our T2D dataset are generated from scripts with several pre-defined templates. We carefully select volunteers with natural language processing backgrounds for drafting templates. Before participating in the annotation process, each annotator was required to undergo a qualification round consisting of a series of test annotations. We illustrate how we generate an instruction to describe one room’s as-
21https://www.mturk.com/
Restricted
Balcony one is located at the northwestern point of the floorplan. It is approximately 50 square feet in size. It can be accessed through the kitchen, bathroom and common room 2.
Balcony 2 is located in the most southern point of the floorplan, just east of the master room. It is roughly 12 feet in length and five feet in width. Access points include the livingroom and master room.
The bathroom is located south of the kitchen. It can be accessed through the east wall of the livingroom, as well as the southern kitchen wall. Common room 2 has acess to the bathroom through its eastern wall. It is the smallest room in the floorplan. At approximately 25 square feet in size, it is just a bit smaller than balcony 1 and half the size of balcony 2.
Common room one is located on the western portion of the floor plan. It can be accessed through common room 2 at the north, the master room at the south and the livingroom from the east. Common room one is about 100 square feet, 10 feet in length and 10 feet in width, making it the second largest room in the floor plan.
Commmon room 2 is just a bit smaller than Common room 1, approximately 90 square feet. It is located just north of Common room 1, with access points including the bathroom at the northeast, balcony at the north, and livingroom at the east.
The kitchen is located in the most northern point of the floorplan. It is roughly 50 square feet, 10 feet in length and 5 feet in width. The bathroom can be accessed at the southern point of the kitchen, the first balcony toward the western kitchen wall and the livingroom at the eastern wall. The kitchen is relatively closest in size to the balconies.
The livingroom is located in the northeastern portion of the floorplan. It is entered through the front entry door, and is approximately 480 square feet. The bathroom, kitchen, common rooms one and two, the master bedroom and second balcony can all be accessed through the livingroom.
The master room is located in the southern end of the floorplan. It is roughly 200 square feet, 10 feet in width and 20 feet in length. The second balcony, livingroom and common room one can be accessed through the master room. The master room is about the size of the two common rooms combined.
Human Instruction Example:
Figure 7: An example of human-written language instructions from the T2D dataset.
pect ratio in Figure 8. We also show a full example of artificial instructions in Figure 9.

heading: D Baseline Generation Samples
text: To better understand and compare different baselines, we provide a case study of generated samples for the same language instructions from all baselines shown in Figure 10. Obj-GAN (Li et al., 2019) has difficulties in capturing salient information from the given language instructions, resulting in generating rooms with incorrect attributes and relationships. One possible reason could be that it does not utilize any pre-trained large language model and thus suffers from understanding the given document-level instructions. CogView (Ding et al., 2021) instead auto-regressively generates the image tokens conditioned on all input instructions with a pre-trained GPT as the backbone. However, the image tokens sampled near the end of the generation show confusing information, resulting in an incomplete design. This is probably because presenting the whole floor plan design as a sequence of image tokens hinders the potential connections among different elements in the floor
plan. Imagen (Saharia et al., 2022b) exhibits its strong ability to generate realistic images in the target domain. However, it also fails to meet various design requirements specified in language instructions, indicating its limitation for design generation under multiple stricter constraints.
Restricted
Restricted
Obj-GAN CogView Imagen Ground Truth
Tell2Design
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
The individual section named Limitations after section 8.
3 A2. Did you discuss any potential risks of your work? The individual section named Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section3, Section5
3 B1. Did you cite the creators of artifacts you used? Section3, Section5
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section Ethics Statement, Section Appendix B
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section Ethics Statement
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section Ethics Statement
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3, Section 4
C 3 Did you run computational experiments? Section 5
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5, Section Appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5, Section Appendix B
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Section 3, Appendix D
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3, Appendix D
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Ethics Statement
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Section 3
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Section 3

heading: Future Work
text: We hope this paper will serve as a foundation and propel future research on the task of the language-guided design generation. Limitations
The proposed T2D dataset has several limitations, which could be addressed in future work. First, it only considers and collects language instructions for the floor plan domain. Future work could extend this language-guided design generation task to other design domains such as documents, mobile UIs, etc. Second, it is limited in the scope of languages where we only collect instructions written in English. Future work could assess the generalizability of the T2D dataset to other languages. Third, although generating floor plan designs from languages exhibit diversity, we do not consider improving generation diversity at this moment. Future works could consider building frameworks that specifically aim at design diversity