Live web prototypes from hand-drawn mockups

Web designers often use physical mockups, such as hand-drawing, to quickly prototype and convey design ideas. Unfortunately, mockups do not reproduce the look and feel of a web page. To achieve this, developers and designers must transform the mockup to the respective HTML and CSS pages. This transformation is often laborious and the final result may not appeal to the client. To address the issues, designers must return to the mockup phase and repeat this cycle. This back and forward is resource and time-consuming. Shortening this cycle would improve the capability of the designers to deliver more accurate and tangible product in a shorter time. The goal of Live Programming is to shorten the code-compile-run cycle of software applications, providing real-time feedback to the developers. The aim of this thesis is to apply the principle of Live Programming to the web design process, by generating the HTML and CSS code necessary from the hand-drawn mockup. This results in a prototype ready to test, where the client and designer can iteratively make changes to the mockup, shortening the feedback time. The transformation of digital mockups to HTML/CSS is something that already exists and is in continuous expansion some implementations even achieve the generation of reactive user interfaces. Notwithstanding the importance and robustness of this well-established tools, we believe that there is room for improvement mainly in the interpretation of hand-drawn mockups. This field is flourishing, and there are already some advancements. Some of the works follow a more traditional approach, using computer vision techniques to extract information from the mockup and on the other side of the spectrum, there are Deep Learning techniques to generate code directly from images, this technique is based on the analogy between generating English textual descriptions from images and generating code from mockups. At the time of this writing, these tools do not seem to make use of semantic annotation to define the behavior of the elements. In light of the aforementioned problem, the proposed solution is the development of software that, based on hand-drawn mockups, extracts the elements and their relative positioning. These elements were then analyzed in an attempt to match their behavior and structure to the mockup. A design language is also produced for the mockups, to facilitate the image processing steps and to enrich the available options with annotations. The development of such a system usually requires a large quantity of data, so the development of a dataset generator is also proposed. The evaluation aimed to assess the dataset generator feasibility, whose data is used for the training of Deep Learning models. When comparing the model trained with only hand-drawn images with another only using synthetic images, similar mean average precision (mAP) performance was measured of 59% and 60% respectively. Using the model that used synthetic data and further fine-tune using hand-drawn data, the mAP value improved to 95%. Therefore, confirming the feasibility of such a generator for atomic element detection when using low quantities of data. We believe this work can have an impact on the development cycle of user interfaces by providing a UI prototype during the mockup phase, thus speeding the development phase.