Patent Description:
A digital experience is any interaction between a user and an organization that is implemented through a digital technology. Companies building software and applications for digital devices often attempt to create a unified digital experience across different digital channels, different applications and different types of devices. Typically, digital experiences are initially created by designers using design tools. The designs are then passed to engineers who specialize in creating code for specific platforms (mobile, web, wearables, AR/VR, etc.) and/or specific software frameworks (Angular, React, Swift, etc.).

Designers may use a variety of different experience design tools, such as Photoshop, Figma, etc. There are currently no unified approaches to converting the outputs from these design tools into deployable code for given choices of platform and framework. There is thus a significant amount of work required to translate the designs into executable code for target platforms and frameworks. In some cases, up to fifty percent of the development effort for a given experience may be spent working on user interface layouts. As the market moves towards full stack developers who prefer working on business logic rather than layouts, there is an even greater need to increase the efficiency of translating designs into deployable code. Moreover, there is a need to efficiently generate code so that different systems (including design tools, software frameworks, and physical platforms) can interoperate more reliably.

There is a need in the art for a system and method that addresses, at least, the shortcomings discussed above.

<NPL> describes converting wireframe from sketches into code.

An intelligent experience generation system and method is disclosed. The intelligent experience generation system and method solves the problems discussed above by automating the process of generating deployable code that implements a digital experience for target platforms (that is, mobile devices, web, wearables, AR/VR, etc.) and frameworks based on input designs. Specifically, the system can automatically convert design images, including wireframes and other graphical designs, for digital experiences (e.g., websites, mobile applications, wearable experiences and VR/AR experiences) into deployable code that implements those digital experiences on a variety of different platforms and for a variety of different frameworks. The system improves on existing low code platforms that allow designers to build a user interface with pre-defined widgets by enabling a <NUM>-<NUM> mapping from input designs to output digital experiences that are not constrained by the available widgets. The system can also take images and/or portable document formatted files (PDFs) as input, rather than customized design file types, thereby allowing the system to incorporate designs from any design tool.

The system may convert wireframe and/or other visual designs into target platform and framework code according to the following process: (<NUM>) render designs on a canvas; (<NUM>) recursively traverse the canvas and read the design pixels; (<NUM>) analyze the pixel maps and recommend indicative patterns (or feature patterns); (<NUM>) transform identified patterns according to target platform standards; (<NUM>) transform platform patterns according to target framework standards; (<NUM>) enhance the patterns according to best industry practices; (<NUM>) display finalized patterns for users; (<NUM>) apply accessibility and disability guidelines; (<NUM>); generate output files; (<NUM>) apply test scaffolding; (<NUM>) output target platform and framework code along with test case scripts.

As part of this process, the system may use artificial intelligence to identify and then classify elements from the input designs. Specifically, identification may be performed using a convolutional neural network (CNN) and the classification may be performed using a Softmax classifier for multi-class classification. Once the system has identified and classified the elements, the code to produce these elements can be automatically generated according to known rules for a target platform and for a target framework. The resulting output can then be tested by users who can provide feedback to help train the system or otherwise iterate on the output.

In accordance with a first aspect there is provided a method, as described in claim <NUM>.

In accordance with a second aspect there is provided a system, as described in claim <NUM>.

In accordance with a third aspect there is provided a non-transitory computer-readable medium, as described in claim <NUM>.

As used herein, a "design image" may comprise any kind of image generated by a designer using design tools. A design image could include wireframes or other graphic designs. Also, a canvas, or "digital canvas", comprises a representation of an image, or of an area of pixels within which an image may be contained.

In some cases, tile maps are fed into a convolutional neural network (CNN), which has been trained to recognize distinct types of feature patterns in images. As used herein, the term "feature pattern" refers to a pattern in an image that can be associated with an element of a design. Thus, a feature pattern may indicate that a given tile map contains a title, an input label, an input box, a menu selector, a photo, or any other element of a design.

In some cases, once the CNN has identified different feature patterns, those features patterns may be classified using a Softmax classifier. The Softmax classifier takes an identified feature pattern and provides probabilities that the feature pattern belongs to a set of known classes. In some cases, the classifier may compare the selected feature pattern with sample/known patterns from a database of known patterns (a pattern factory).

In another aspect, a system for automatically generating code to be executed on a target digital platform includes a device processor, and a non-transitory computer readable medium storing instructions that are executable by the device processor to implement: (<NUM>) a canvas rendering engine that renders an input design image on a digital canvas; (<NUM>) a design parser that reads pixels in the design image and parses the pixels into tile sets; (<NUM>) a pattern identification and classification module that can convert the tile sets into tile maps, further comprising a neural network that identifies feature patterns in the tile maps and a classifier that classifies the feature patterns identified by the neural network; (<NUM>) a platform pattern transformation module that transforms the identified feature patterns into platform specific programming objects; and (<NUM>) an output generator that outputs deployable code for the target digital platform, wherein the code reproduces the feature patterns on the target platform.

In another aspect, a non-transitory computer-readable medium storing software includes instructions executable by one or more computers which, upon such execution, cause the one or more computers to: (<NUM>) receive a design image; (<NUM>) render the design image on a digital canvas and read pixels of the design image; (<NUM>) automatically generate a tile map from the pixels of the design image upon reading the design image; (<NUM>) use a neural network to identify a feature pattern in the tile map; (<NUM>) classify the feature pattern in the tile map; and (<NUM>) use the classified feature pattern to automatically generate code, wherein the code reproduces the feature pattern in the tile map on the target platform.

Other systems, methods, features, and advantages of the disclosure will be, or will become, apparent to one of ordinary skill in the art upon examination of the following figures and detailed description. It is intended that all such additional systems, methods, features, and advantages be included within this description and this summary, be within the scope of the disclosure, and be protected by the following claims.

While various embodiments are described, the description is intended to be exemplary, rather than limiting and it will be apparent to those of ordinary skill in the art that many more embodiments and implementations are possible that are within the scope of the embodiments. Although many possible combinations of features are shown in the accompanying figures and discussed in this detailed description, many other combinations of the disclosed features are possible. Any feature or element of any embodiment may be used in combination with or substituted for any other feature or element in any other embodiment unless specifically restricted.

This disclosure includes and contemplates combinations with features and elements known to the average artisan in the art. The embodiments, features and elements that have been disclosed may also be combined with any conventional features or elements to form a distinct invention as defined by the claims. Any feature or element of any embodiment may also be combined with features or elements from other inventions to form another distinct invention as defined by the claims. Therefore, it will be understood that any of the features shown and/or discussed in the present disclosure may be implemented singularly or in any suitable combination. Accordingly, the embodiments are not to be restricted except in light of the attached claims and their equivalents. Also, various modifications and changes may be made within the scope of the attached claims.

A system and method for intelligently and automatically generating deployable code that implements digital experiences for target platforms and frameworks based on images or other graphical inputs is disclosed. The digital experiences can include graphics, inputs (such as menus and forms), and other elements associated with software applications running on various digital platforms. The system and method leverage artificial intelligence to automatically identify and classify elements of a design as feature patterns. The intelligent system can then automatically generate code for target platforms and frameworks that reproduce the feature patterns and thereby implement a digital experience for the users. Target platforms may include web platforms, mobile platforms (such as mobile applications/browsers), wearable platforms (such as smart watches), and extended reality platforms (which includes augmented reality (AR), virtual reality (VR), and/or combinations of AR/VR). The code may also be automatically generated for use with target frameworks. Target frameworks can include, for example, server-side web frameworks such as Ruby on Rails and Django, and client-side web frameworks such as Angular and React, or mobile application frameworks such as Android SDK or Swift.

<FIG> is a schematic view of a system for automatically converting experience designs into deployable code that can be run on a variety of platforms. The deployable code implements a digital experience on the target platform that corresponds to the experience design. Specifically, experience designs <NUM> are fed as inputs into an intelligent experience generation engine <NUM>. Experience designs <NUM> may comprise any outputs of a design tool or other graphical interface. For example, experience designs <NUM> could comprise wireframes or other visual designs. The format of the experience designs could be any image based or graphical based format, such as JPEG, SVG, PDF, PNG or GIF. Other format types include formats associated with specific design tools, such as the adobe photoshop format or the adobe illustrator format. In the exemplary embodiment, experience designs may be PDFs or other image files. However, in some cases, intelligent experience generation engine <NUM> could include provisions for converting files in other formats (such as the adobe photoshop format) into PDF or image files (such as JPEG).

Intelligent experience generation engine <NUM>, also referred to simply as engine <NUM>, takes the PDF or image files for an experience design and outputs code that can be deployed across multiple different target platforms so as to provide a variety of different digital experiences associated with the experience design. For example, engine <NUM> may output modern web components <NUM> (such as HTML files, JavaScript files, and/or various web framework components) that can be used to implement a web experience <NUM>. Also, engine <NUM> may output screen layouts <NUM> (such as responsive layouts or native android/ios screen layouts that are important for mobile design) that can be used to implement a mobile experience <NUM>. Also, engine <NUM> may output wearable layouts <NUM> (which are generally highly constrained in size as compared to mobile and web layouts) that can be used as part of a wearable experience <NUM>. Also, engine <NUM> may output AR/VR scenes <NUM> that can be used as part of an extended reality experience <NUM>.

Each of these experiences may be consumed by software architects, software developers, usability testers, and/or end users <NUM>. Some of these people may participate in a feedback cycle. This feedback cycle is used to provide feedback <NUM> to intelligent experience generation engine <NUM> to refine the pattern recognition abilities of the system.

Engine <NUM> may be configured with one or more processors <NUM> and memory <NUM>. Memory <NUM> may comprise a non-transitory computer readable medium. Instructions stored within memory <NUM> may be executed by the one or more processors <NUM>. More specifically, memory <NUM> may store software modules and other software components, as well as data, which facilitates translating experience designs into deployable code that implements digital experiences.

<FIG> is a schematic view of an architecture for intelligent experience generation engine <NUM>, according to one embodiment. As already discussed, engine <NUM> may receive input including designs <NUM>. Additionally, engine <NUM> receives information about the target platform and framework <NUM>. Optionally, engine <NUM> may receive information about the target industry <NUM>, which may inform how the designs are translated into digital experiences. Moreover, in some cases, engine <NUM> may be in communication with an external component that may be consulted for accessibility and compliance guidelines <NUM>.

Engine <NUM> automatically generates output <NUM>. The output is comprised of code that is specific to a selected target platform and framework. In some cases, engine <NUM> may generate code for multiple different platform and framework combinations. Output <NUM> is then used to generate experiences for users <NUM> who can, optionally, provide feedback.

Engine <NUM> may be comprised of a plurality of different components. These may include software modules for performing particular operations, as well as data storage components (for example, databases). For example, engine <NUM> can include a canvas rendering engine <NUM>, a design parser <NUM>, a pattern identification and classification module <NUM>, a platform pattern transformation module <NUM>, a framework pattern transformation module <NUM>, an industry pattern enhancer <NUM>, a feedback analyzer <NUM>, an accessibility and disability validator <NUM>, an output generator <NUM>, and a test scaffolding module <NUM>. Additionally, engine <NUM> may include various data storage systems, such as a pattern factory <NUM>, a platform pattern repository <NUM>, a framework components repository <NUM>, and an industry pattern best practices store <NUM>. Each component may be configured with one or more sub-modules or sub-components that facilitate converting the input into deployable code, as described in further detail below.

<FIG> is a schematic view of a process <NUM> for automatically converting input designs into code that can be deployed on a target platform and for a target framework to implement a digital experience. Some of these steps may be performed by one or more of the modules or components described above and shown in <FIG>.

The exemplary process starts with inputs <NUM>. These may include wireframe and/or other visual designs, the target platform(s) and/or framework(s), as well as the target industry. Based on this input data, engine <NUM> renders the designs on a canvas for further decoding in step <NUM>. Specifically, using canvas rendering engine <NUM>, engine <NUM> may receive command line inputs from the user and covert the inputs into configuration objects with details for: sources files associated with the experience design, the output destination for the code, target platforms, target frameworks, and a target industry (if applicable). Canvas rendering engine <NUM> may then make a network call to any associated cloud interfaces to retrieve the image files and any necessary rulesets to be distributed to other components of engine <NUM>. Canvas rendering engine <NUM> further reads in source image files and prepares an image object and any metadata based on the dimensions and resolutions of the design images. Canvas rendering engine <NUM> may also read in any PDF files and parse those files to extract an image object and any metadata. The source files can then be iterated over and one or more HTML canvases can be constructed using a headless browser controller in a browser. The design images are then rendered in the canvas.

Once the canvases have been created, engine <NUM> may recursively traverse each canvas and read in the pixels from the design images in step <NUM>. This step is performed by design parser <NUM>. Specifically, design parser <NUM> parses each image object in the canvas into individual pixel elements. Then, design parser <NUM> creates a tile set by iterating through each pixel using 2D traversal. Here, each "tile" in the tile set is comprised of a group of pixels having a predefined set of rows and columns. The size of the tile sets could be set according to the desired degree of granularity and accuracy. Once the tile sets have been prepared, design parser <NUM> assembles the individual tiles into a 2D array that represents the original design image.

After the tile sets for each image have been created, engine <NUM> may analyze the tile sets to identify and classify portions of the image in step <NUM>. Specifically, pattern identification and classification module <NUM> receives a 2D array of the tile set assembled by design parser <NUM>. Using rulesets that have been previously retrieved, module <NUM> iterates through the tile set to create multiple tile maps. A tile map (or "pixel map") may correspond to one or more tiles that have been grouped together. Once a set of tile maps have been created, module <NUM> analyzes each tile map to identify feature patterns. The term "feature pattern" refers to a pattern in an image that can be associated with an element of a design. Thus, a feature pattern indicates that a given tile map contains a title, an input label, an input box, a menu selector, a photo, or any other element of a design.

In some embodiments, feature patterns may be identified using a deep neural network (DNN). The DNN may comprise different kinds of layers including convolutional layers, pooling layers, and/or fully connected layers. In one embodiment, the DNN may comprise at least one convolutional layer that feeds into at least one pooling layer. Additionally, the at least one pooling layer could feed into a series of two or more fully connected layers.

Once module <NUM> has identified different feature patterns, those features are classified as in step <NUM>. Classification is performed using a Softmax classifier. The Softmax classifier takes an identified feature pattern and provides probabilities that the feature pattern belongs to a set of known classes. In some cases, the classifier may compare the selected feature pattern with sample/known patterns from pattern factory <NUM>.

In order to better illustrate the feature identification and classification process, <FIG> depicts a schematic view of a process <NUM> that may be performed by module <NUM>. As seen in <FIG>, module <NUM> receives a tile set <NUM> and constructs an associated set of tile maps, which may correspond to groups of tile sets. Each image tile map <NUM> is then analyzed by a convolutional neural network <NUM>. As described above, the convolutional neural network may also incorporate pooling layers as well as fully connected layers as part of its architecture.

As an example, a tile map <NUM> is provided as input to CNN <NUM>. CNN <NUM> identifies tile map <NUM> as an "input element". Softmax classifier <NUM> may receive both tile map <NUM> and the "input element" identifier. Based on this information, classifier <NUM> compares tile map <NUM> with known sample input elements and estimates a probability that tile map <NUM> corresponds to a particular type of UI element. For example, classifier <NUM> gives a high probability (<NUM> or <NUM>%) to tile map <NUM> being a "label" (for an input element). Based on this classification, tile map <NUM> is identified with an output pattern map <NUM> that will be used to automatically generate code.

The table <NUM> shown in <FIG> further illustrates how tile maps can be identified and classified. For example, in row <NUM> of the table in <FIG>, several different tile maps are presented. In this example tile map <NUM> is a fragment of an actual design element. Specifically, tile map <NUM> namely part of the first name label on a web form. By contrast, tile map <NUM> corresponds to the complete label "first name", and tile map <NUM> corresponds to the complete label "last name. " Having been identified by the neural network, the feature pattern of each tile map is then classified by the classifier. The probabilities for each feature pattern are given in the bottom four rows of the table. For example, tile map <NUM>, which is identified as a "background" element, has a <NUM>% probability of being a background element, according to the classifier. Likewise, tile map <NUM> and tile map <NUM> each have a <NUM>% probability of being labels. Therefore, tile map <NUM> and tile map <NUM> are selected (that is, retained for further processing). By contrast, tile map <NUM> has only a <NUM>% chance of being a label, and no more than a <NUM>% chance of being any other known element. Therefore, element <NUM> is discarded by the engine and not used to generate elements in later steps.

Referring back to <FIG>, after generating a finalized set of patterns (via identification and classification) in step <NUM> and step <NUM>, engine <NUM> may transform the identified patterns into the appropriate coding elements for the target platform in step <NUM>. Specifically, platform pattern transformation module <NUM> may transform feature patterns into core elements for a target platform. Here, a "core element" refers to a class, object, or other programming element used to generate a particular digital element in a given platform. For example, if the identified pattern is an input text element, the associated core element for a web platform would be the HTML element "<input type='text'>". In some cases, the core elements for each pattern can be retrieved from platform pattern repository <NUM>.

In step <NUM>, engine <NUM> may transform the platform pattern to target framework patterns. Specifically, framework pattern transformation module <NUM> may convert the platform specific core elements identified in step <NUM> into framework specific elements for the target platform. For example, HTML, CSS and JavaScript code could be reorganized (or transformed) into standard framework components that implement the digital experience using known rules and strategies.

In step <NUM>, the framework specific patterns can be enhanced according to best practices that may be industry specific. Specifically, industry pattern enhancer <NUM> may retrieve best practices from industry best practices storage <NUM>. These best practices may provide guidance for implementing the framework patterns so that the resulting code is in compliance with security standards, aesthetic standards, and/or any other kinds of standards associated with a particular industry.

In step <NUM>, engine <NUM> may display the finalized patterns for user feedback. As an example, <FIG> illustrates a sample output. In this output, experience elements are superimposed over the original design image <NUM> so that a user can easily see how well the automatically generated digital experience matches the experience design. For clarity, only some digital experience elements are shown.

As an example, design image <NUM> includes a first design box <NUM> (for inputting a first name) and a second design box <NUM> (for inputting a last name). In this case, first input box <NUM> is overlaid onto first design box <NUM>. As can be seen, first input box <NUM> does not have the correct size or alignment to match first design box <NUM>. By contrast, second input box <NUM> substantially matches second experience box <NUM>. Thus, a user reviewing the output of step <NUM> may provide feedback that indicates that first input box <NUM> is incorrect. The system may then iterate over the design until this error is corrected.

Referring back to the process in <FIG>, in step <NUM>, engine <NUM> may apply accessibility and disability guidelines. Specifically, accessibility and disability validator <NUM> may iterate through the elements of the output to ensure compliance with accessibility and disability guidelines. In some cases, as part of this step, validator <NUM> may consult accessibility and compliance guidelines <NUM> (shown in <FIG>).

In step <NUM> output files can be generated. In step <NUM> the system may also output test scaffolding files that can be used to test the automatically generated code against various templates. The outputs <NUM> may therefore include both target platform and framework code <NUM>, as well as test case scripts <NUM>.

<FIG> is a schematic view of one embodiment of output generator <NUM>. Output generator <NUM> may comprise various modules for generating the deployable code for implementing unified experiences across a variety of different digital platforms. It may be appreciated that output generator <NUM>, shown in <FIG>, could include some or all of the same modules and other components as output generator <NUM>.

As shown in <FIG>, output generator <NUM> may receive layout map <NUM> as input. Layout map <NUM> may comprise the target framework patterns (themselves converted from the target platform patterns) necessary to build the final layout for deployment on one or more platforms.

Information from layout map <NUM> is fed into a layout map iterator <NUM>. Layout map iterator <NUM> iterates through each element in layout map <NUM>. Each element may be identified according to its metadata. The elements may then be handed over to an appropriate interface for generating the output code. For example, a core element interface <NUM> provides the necessary code generator function definitions for the core elements. A component generator interface <NUM> provides the necessary code generator function definitions for components that are not already identified as reusable components or industry ready components. A composite component generator <NUM> interface provides the necessary code generator function definitions for the composite components that are not already identified as reusable components or industry ready components. A styling component generator interface <NUM> provides the necessary code generator function definitions for the styling generation of core elements, components or composite elements. A reusable component repository interface <NUM> provides the necessary interfaces and implementations to pull the code from the existing reusable components repository and the necessary binding to the target output code. An industry component repository interface <NUM> provides the necessary interfaces and implementations to pull the code from the existing industry components repository and the necessary binding to the target output code. In some cases, interface <NUM> may retrieve information from a component repository <NUM>.

To generate code for different digital experiences, output generator <NUM> may include a web experience output generator <NUM>, a native mobile and wearable experience output generator <NUM>, and an AR/VR experience output generator <NUM>. For example, web experience output generator <NUM> includes provisions to implement classes for various web platform components. Generator <NUM> may further include an HTML template generator <NUM> to provide the HTML template generation capabilities as per the standard W3C definitions. Generator <NUM> may also include a CSS generator <NUM> to provide the CSS styling capabilities as per the standard CSS3 specifications. Generator <NUM> may also include an SPA generator interface <NUM> to provide the necessary interfaces to integrate with existing CLI based open source tools for single page application generation and component creation such as "ng CLI", "createreactapp CLI", and "Vue CLI". In some cases, generator <NUM> may communicate with an open source CLI interface <NUM>.

Native mobile and wearables experience output generator <NUM> may include provisions to implement classes for the various native mobile and wearable platform components. For example, generator <NUM> may include an XML layout interface <NUM> for generating native UI layouts, a themes generator <NUM> for applying styling to the layouts, and a native SDK interface <NUM> for project and component creation and for binding the XML layouts. In some cases, generator <NUM> may communicate with an android/ios SDK interface <NUM>.

ARVR experience output generator <NUM> may include provisions to implement classes for the various ARVR scene definitions. These may have specific implementations to generate code for specific elements. Generator <NUM> may include a unity output generator <NUM>, which may interface with unity based scripting toolsets for the AR/VR scene generation (for example, with a unity SDK interface <NUM>).

Using the exemplary system and method, experience designs in the form of images or PDFs can be automatically converted into code to implement digital experiences on a variety of platforms. The system and method may therefore provide significant time savings for developers, allowing them to focus on the business logic of applications rather than building user layouts from designs.

Claim 1:
A method for automatically generating code to be executed on a target digital platform, comprising:
receiving a graphical design image as a source image file;
constructing an HTML canvas using a headless browser controller in a browser from the source image file;
rendering the design image on the HTML canvas and reading pixels of the graphical design image by a design parser (<NUM>) parsing an image object in the HTML canvas into individual pixel elements;
the design parser (<NUM>) creating a tile set comprising a group of individual pixel elements having a predefined set of rows and columns;
automatically generating an arrangement of tiles as a tile map (<NUM>) in a 2D array of tiles from the tile set;
using a neural network (<NUM>) to identify a feature pattern in the tile map (<NUM>), wherein the feature pattern is a pattern of the image associated with an element of the design;
classifying the feature pattern in the tile map (<NUM>) using a Softmax classifier (<NUM>);
using the classified feature pattern to automatically generate code, wherein the code reproduces the feature pattern in the tile map (<NUM>) on the target digital platform; and
outputing test scaffolding files comprising test case scripts (<NUM>) for testing the automatically generated code.