English:
[0001] Present invention is related to a system and a method for development and deployment of flexible, dynamically editable and load-balanced GUI in a connected real time system.
[0002] Typically, the UX designers create all digital assets on their PC/Mac which includes screen flows and contents (images and texts) using digital content creation software tools like Sketch, Photoshop, etc. However, even after creating the complete visualization they need to create written and diagrammatic specification/requirements documents so that it could be converted into a software that can be executed on a target device with appropriate performance and load balancing. This involves a lot of effort to understand complex GUI behaviors using the specification documentation in order to convert the specification to target hardware specific visualization and performance. This results in a lot of iterations, delays, visual defects and performance defects.
[0003] Presently most of the software products are capable of performing basic image and text import from GUI designs and generate GUI software for these basic screens. However, these products do not generate and provide live editing of the GUI contents in real time. The products are also not capable of monitoring the computing resource load on the connected real time system and generate GUI software that is capable of balancing the computing resource load.
[0004] Moreover, there are a few software products are available to convert digital assets like images and text from GUI designs to partial GUI software components or basic HTML pages or GUI prototypes. However these software need to be ported to the target hardware manually and later need to be tuned for performance by developers. Besides, this process needs to be repeated every time there is a change in the GUI screen flows or screen visualization or when the input sources change.
[0005] According to an US application numbered US6496202B, a method and apparatus for generating a graphical user interface. This patent provides a design using which a GUI can change its visualization depending on how the user interacts with the application in the field. For example, when a user clicks a button, a part of the screen/fragment/control can be switched off and a new screen/fragment/control can be added automatically. However someone needs to explicitly decide what the behavior shall be when the event happens and once this is specified the design helps in generating the GUI that satisfies the new requirements. However, the disclosed system do not allow the GUI behaviors or specification itself to be edited in real time on the connected real time device. It also does not optimize the GUI based on the performance and load on the real time system.
[0006] Hence, there is a need of a solution to capture the inputs from the UX designers from various mediums, tools and formats and then generate machine understandable GUI specification that is capable of being directly interpreted and executed on the connected real time system without any manual involvement.
[0008] Figure 1 is a block diagram illustrating a system for deployment of dynamically editable GUI on a connected real-time device, according to the aspects of the present invention; and
[0009] FIG. 2 is an example process for deployment of dynamically editable GUI on a connected real-time device using the system of FIG. 1 , according to the aspects of the present technique.
[0010] Fig. 1 illustrates overall structure and components involved in a system 100, in which example embodiments of the present invention may be deployed. The system 100 is adapted to automatically deploy the machine executable graphical user interface (hereinafter“GUI”) specifications on a connected real time device. The system 100 may be deployed in various environments. For example, the system 100 can be deployed on a cloud or a server which can then service the requests/inputs from several clients. The system 100 includes an input module 102, a GUI specification generator 104, a GUI configurator 106, a real-time module 108, a rendering module 116 and output module 118. Each component is described in further detail below.
[0011] The input module 102 is configured to receive a plurality of GUI inputs from a user, herein UX designer. In one embodiment, the plurality of GUI inputs may be captured from several mediums such as images via camera, screenshots, frame- grabber, video, audio, digital content creation tools like Photoshop, Sketch, and the like. In this embodiment, the GUI inputs may be live feed or recorded playback.
[0012] The input module 102 is further configured to convert the plurality of GUI inputs to digital formats, data and meta-data which are relevant for GUI development. In addition, the plurality of GUI inputs are processed to identify the building blocks of a GUI such as GUI screen flows, GUI layouts, GUI contents and the like. In one example, pattern matching, image comparisons, context aware content recognition, machine learning techniques may be used to perform the identification of the building blocks of the GUI. However, a variety of other identification techniques may be envisaged.
[0014] The GUI Configurator 106 is configured to inject performance load balancing parameters and configuration data along with the GUI configuration data. The GUI Configurator 106 is further configured to parse the digital asset data and meta-data related to the GUI. In one embodiment, after the parsing is done, the GUI configurator 106 is further configured to enable the UX designer/user to edit the GUI flows, layouts and contents in the connected real time system and see the result on an output module 118 in real time.
[0015] In an alternate embodiment, the GUI configurator 106 is integrated to a content management system (CMS) server. Further, the GUI configurator 106 configured to receive several dynamic updates from a content management system (CMS) server for the latest digital assets.
[0016] The real-time module 108 configured to automatically deploy the machine executable GUI specification on the connected real time system and edit the GUI inputs in real time. In an embodiment the real-time module 108 may be deployed in various environments. For example, the real-time module 108 can be deployed over, websites, desktop, PC’s, MAC or the like. The real-time module 108 includes a storage module 110, load balancer 112 and a loading engine 114. Each component is described in further detail below.
[0017] The storage module 110 is configured to store the machine understandable GUI specification, generated by the GUI specification generator 104. In one embodiment, the machine understandable GUI specification includes GUI screen flows, layouts and contents. The storage module 100 is configured to store the machine understandable GUI specification in the form of XMLs, binaries, configuration parameters, tables, OpenGL/WebGU/Vulkan/OpenVG/2D graphics lib invocations, and the like.
[0018] The load balancer 112 is configured to collect real time computing resource loads from the connected real time system. The load balancer 112 runs on the real time system and continuously keeps monitoring the load.
[0019] The loading engine 114 is configured to ensure that the graphical user interface specification is loaded from the storage module (110) and forwarded to a rendering module (116). The rendering module 116 is configured to execute the generated machine understandable GUI specification on a real time system. In one embodiment, when the GUI specifications such as screen flow, layouts and contents are executed on the rendering module 116, the load is monitored by the load balancer 112 and sent to the GUI Configurator 106 in real time as a load balancing configuration. The GUI Configurator 106 uses the load balancing configuration to optimize the GUI configuration Flow, layout and contents. In one example, the computing resource load on the connected real time system is monitored and its usage is evaluated in real time to derive the optimal load balancing strategy.
[0020] In another embodiment, the GUI Configurator 106 is interactively connected with the load balancer 112 on the connected real time device. Based on the load balancing configuration data received from the load balancer 112, the GUI Configurator 106 injects performance load balancing parameters and configuration data along with the GUI flow, layouts and content configuration data. Further, this data flows to the GUI Specification generator 104 which uses the configuration data to generate a load balanced application. In one embodiment, This load balanced GUI configuration is then sent to the GUI Specification generator which in turn generates the machine understandable GUI specification that is then stored on the storage module 110 of the connected real time system.
[0021] FIG. 2 is an example process 200 for deployment of dynamically editable GUI on a connected real-time device using the system 100 of FIG. 1 , according to the aspects of the present technique.
[0022] At step 202, a plurality of GUI inputs are received, to identify the building blocks of the graphical user interface. In an embodiment, the plurality of GUI inputs may be captured from several mediums such as images via camera, screenshots, frame- grabber, video, audio, digital content creation tools like Photoshop, Sketch, and the like. In some embodiments, the GUI inputs are accessed from other locations such as from an offline image repository, cloud storage and so forth. In an embodiment, the GUI inputs may be live feed or recorded playback.
[0023] At step 204, the plurality of GUI inputs are converted into digital format for use by the graphical user interface. The plurality of GUI inputs are processed to identify the building blocks of a GUI such as GUI screen flows, GUI layouts, GUI contents and the like. In one example, pattern matching, image comparisons, context aware content recognition, machine learning techniques may be used to perform the identification of the building blocks of the GUI. However, a variety of other identification techniques may be envisaged.
[0024] At step 206, the digital graphical user interface inputs are parsed and machine understandable graphical user interface specification is generated from the plurality of GUI inputs. At step 208, the graphical user interface behavior is edited in real time, on the connected real time system. After the parsing is done, the GUI configurator 106 of Fig. 1 is configured to enable the UX designer/user to edit the GUI flows, layouts and contents in the connected real time system and see the result on an output module 118 in real time.
[0025] Portions of the example embodiments and corresponding detailed description may be presented in terms of software, or algorithms and symbolic representations of operation on data bits within a computer memory. These descriptions and representations are the ones by which those of ordinary skill in the art effectively convey the substance of their work to others of ordinary skill in the art. An algorithm, as the term is used here, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.
[0026] The system(s)/apparatus(es), described herein, may be realized by hardware elements, software elements and/or combinations thereof. For example, the devices and components illustrated in the example embodiments of inventive concepts may be implemented in one or more general-use computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor or any device which may execute instructions and respond. A central processing unit may implement an operating system (OS) or one or more software applications running on the OS. Further, the processing unit may access, store, manipulate, process and generate data in response to execution of software. It will be understood by those skilled in the art that although a single processing unit may be illustrated for convenience of understanding, the processing unit may include a plurality of processing elements and/or a plurality of types of processing elements. For example, the central processing unit may include a plurality of processors or one processor and one controller. Also, the processing unit may have a different processing configuration, such as a parallel processor.
[0027] The methods according to the above-described example embodiments of the inventive concept may be implemented with program instructions which may be executed by computer or processor and may be recorded in computer-readable media. The media may also include, alone or in combination with the program instructions, data files, data structures, and the like. The program instructions recorded in the media may be designed and configured especially for the example embodiments of the inventive concept or be known and available to those skilled in computer software. Computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as compact disc-read only memory (CD-ROM) disks and digital versatile discs (DVDs); magneto-optical media; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like. Program instructions include both machine codes, such as produced by a compiler, and higher level codes that may be executed by the computer using an interpreter. The described hardware devices may be configured to execute one or more software modules to perform the operations of the above-described example embodiments of the inventive concept, or vice versa.
[0028] It should be home in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, or as is apparent from the discussion, terms such as“processing” or“computing” or“calculating” or“determining” of“displaying” or the like, refer to the action and processes of a computer system, or similar electronic computing device/hardware, that manipulates and transforms data represented as physical, electronic quantities within the computer system’s registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
[0029] It should be understood that embodiments explained in the description above are only illustrative and do not limit the scope of this invention. Many such embodiments and other modifications and changes in the embodiment explained in the description are envisaged. The scope of the invention is only limited by the scope of the claims.

中文:
[0001]本发明涉及用于在连接的实时系统中开发和部署灵活、可动态编辑且负载平衡的GUI的系统和方法。
[0002]通常,UX设计者使用数字内容创建软件工具(如Sketch、Photoshop等)在其PC/Mac上创建所有数字资产,这包括屏幕流和内容(图像和文本)。然而,即使在创建了完整的可视化之后,他们也需要创建书面的和图解的规范/要求文档,使得它可以被转换成可以在具有适当性能和负载平衡的目标设备上执行的软件。这涉及到使用规范文档来理解复杂的GUI行为的大量努力,以便将规范转换为目标硬件特定的可视化和性能。这导致了大量的迭代、延迟、视觉缺陷和性能缺陷。
[0003]目前,大多数软件产品能够执行从GUI设计的基本图像和文本导入,并为这些基本屏幕生成GUI软件。然而,这些产品不实时生成和提供GUI内容的实况编辑。产品也不能监视在连接的实时系统上的计算资源负载,并且生成能够平衡计算资源负载的GUI软件。
[0004]此外,存在可用于将来自GUI设计的如图像和文本之类的数字资产转换为部分GUI软件部件或基本HTML页面或GUI原型的几个软件产品。然而,这些软件需要手动移植到目标硬件,并且随后需要由开发人员针对性能进行调谐。此外,每当GUI屏幕流或屏幕可视化存在改变或者当输入源改变时,都需要重复该过程。
[0005]根据编号为US6496202B的美国申请,一种用于生成图形用户接口的方法和装置。该专利提供了一种设计,使用该设计,GUI可以取决于用户如何在现场中与应用交互来改变其可视化。例如,当用户点击按钮时,可以关闭屏幕/片段/控件的一部分,并且可以自动添加新的屏幕/片段/控件。然而,当事件发生时,需要有人明确地决定行为应该是什么,并且一旦这被指定,设计就帮助生成满足新要求的GUI。然而,所公开的系统不允许在连接的实时设备上实时编辑GUI行为或规范本身。它也没有基于实时系统上的性能和负载来优化GUI。
[0006]因此,存在对一种解决方案的需要,该解决方案从各种介质、工具和格式捕获来自UX设计者的输入,并且然后生成机器可理解的GUI规范,该机器可理解的GUI规范能够在没有任何手动参与的情况下在连接的实时系统上直接解释和执行。
[0008]图1是根据本发明的各方面的用于在连接的实时设备上部署动态可编辑GUI的系统的框图;以及
[0009]图2是根据本技术的方面的用于使用图1的系统在连接的实时设备上部署动态可编辑GUI的示例过程。
[0010]图1图示了系统100中涉及的总体结构和部件,其中可以部署本发明的示例实施例。系统100适于在连接的实时设备上自动部署机器可执行图形用户接口(下文为“GUI”)规范。系统100可以部署在各种环境中。例如,系统100可以部署在云或服务器上,该云或服务器然后可以服务于来自若干个客户端的请求/输入。系统100包括输入模块102、GUI规范生成器104、GUI配置器106、实时模块108、渲染模块116和输出模块118。下面进一步详细描述每个部件。
[0011]输入模块102被配置为从用户(本文为UX设计者)接收多个GUI输入。在一个实施例中,多个GUI输入可以从若干个介质捕获,所述若干个介质诸如是经由相机的图像、截屏、帧抓取器、视频、音频、如Photoshop、Sketch的数字内容创建工具等。在该实施例中,GUI输入可以是实况转播或记录回放。
[0012]输入模块102进一步被配置为将多个GUI输入转换为与GUI开发相关的数字格式、数据和元数据。此外,处理多个GUI输入以标识GUI的构建块,诸如GUI屏幕流、GUI布局、GUI内容等。在一个示例中,可以使用模式匹配、图像比较、上下文感知内容识别、机器学习技术来执行GUI的构建块的标识。然而,可以设想多种其他标识技术。
[0014]GUI配置器106被配置为将性能负载平衡参数和配置数据连同GUI配置数据一起注入。GUI配置器106进一步被配置为解析与GUI相关的数字资产数据和元数据。在一个实施例中,在解析完成之后,GUI配置器106进一步被配置为使得UX设计者/用户能够编辑在连接的实时系统中的GUI流、布局和内容,并且实时在输出模块118上看到结果。
[0015]在替代实施例中,GUI配置器106集成到内容管理系统(CMS)服务器中。此外,GUI配置器106被配置为从内容管理系统(CMS)服务器接收针对最新数字资产的若干个动态更新。
[0016]实时模块108被配置为在连接的实时系统上自动部署机器可执行GUI规范,并实时编辑GUI输入。在实施例中,实时模块108可以部署在各种环境中。例如,实时模块108可以部署在网站、桌面、PC、MAC等上。实时模块108包括存储模块110、负载平衡器112和加载引擎114。下面进一步详细描述每个部件。
[0017]存储模块110被配置为存储由GUI规范生成器104生成的机器可理解的GUI规范。在一个实施例中,机器可理解的GUI规范包括GUI屏幕流、布局和内容。存储模块100被配置为以XML、二进制、配置参数、表格、OpenGL/WebGL/Vulkan/OpenVG/2D图形库调用等形式存储机器可理解的GUI规范。
[0018]负载平衡器112被配置为从连接的实时系统收集实时计算资源负载。负载平衡器112运行在实时系统上,并持续保持监视负载。
[0019]加载引擎114被配置为确保从存储模块(110)加载图形用户接口规范,并将其转发至渲染模块(116)。渲染模块116被配置为在实时系统上执行所生成的机器可理解的GUI规范。在一个实施例中,当诸如屏幕流、布局和内容的GUI规范在渲染模块116上执行时,负载由负载平衡器112监视,并作为负载平衡配置被实时发送到GUI配置器106。GUI配置器106使用负载平衡配置来优化GUI配置流、布局和内容。在一个示例中,监视在连接的实时系统上的计算资源负载,并实时评估其使用,以导出最佳负载平衡策略。
[0020]在另一实施例中,GUI配置器106与连接的实时设备上的负载平衡器112交互地连接。基于从负载平衡器112接收的负载平衡配置数据,GUI配置器106将性能负载平衡参数和配置数据连同GUI流、布局和内容配置数据一起注入。此外,该数据流向GUI规范生成器104,GUI规范生成器104使用配置数据来生成负载平衡的应用。在一个实施例中,该负载平衡的GUI配置然后被发送到GUI规范生成器,该GUI规范生成器进而生成机器可理解的GUI规范,该机器可理解的GUI规范然后被存储在连接的实时系统的存储模块110上。
[0021]图2是根据本技术的方面的用于使用图1的系统100在连接的实时设备上部署动态可编辑GUI的示例过程200。
[0022]在步骤202处,接收多个GUI输入以标识图形用户接口的构建块。在实施例中,多个GUI输入可以从若干个介质捕获,所述若干个介质诸如是经由相机的图像、截屏、帧抓取器、视频、音频、如Photoshop、Sketch的数字内容创建工具等。在一些实施例中,从其他位置、诸如从离线图像储存库、云存储装置等等访问GUI输入。在实施例中,GUI输入可以是实况转播或记录回放。
[0023]在步骤204处,将多个GUI输入转换成数字格式,以供图形用户接口使用。处理多个GUI输入以标识诸如GUI屏幕流、GUI布局、GUI内容等之类的GUI的构建块。在一个示例中,可以使用模式匹配、图像比较、上下文感知内容识别、机器学习技术来执行GUI的构建块的标识。然而,可以设想多种其他标识技术。
[0024]在步骤206处,解析数字图形用户接口输入,并从多个GUI输入生成机器可理解的图形用户接口规范。在步骤208处,在连接的实时系统上实时编辑图形用户接口行为。在解析完成之后,图1的GUI配置器106被配置为使得UX设计者/用户能够编辑在连接的实时系统中的GUI流、布局和内容,并且实时在输出模块118上看到结果。
[0025]部分示例实施例及对应的详细说明可以依据软件或算法以及计算机存储器内在数据位上操作的符号表示来呈现。这些描述和表示是本领域普通技术人员据此向本领域其他普通技术人员有效传达其工作实质的描述和表示。如这里使用的以及如一般使用的术语“算法”被认为是导致期望结果的自洽的步骤序列。步骤是需要对物理量进行物理操控的步骤。通常,尽管不是必须的,但这些量采取能够被存储、传输、组合、比较和以其他方式操控的光学、电或磁信号的形式。主要出于通用的原因,将这些信号称为位、值、元素、符号、字符、术语、数字等有时被证明是方便的。
[0026]本文所描述的(一个或多个)系统/(一个或多个)装置可以通过硬件元件、软件元件和/或其组合实现。例如,本发明构思的示例实施例中说明的设备和部件可以在一个或多个通用计算机或专用计算机中实现,所述一个或多个通用计算机或专用计算机诸如是处理器、控制器、算术逻辑单元(ALU)、数字信号处理器、微型计算机、现场可编程阵列(FPA)、可编程逻辑单元(PLU)、微处理器或可以执行指令并进行响应的任何设备。中央处理单元可以实现操作系统(OS)或在OS上运行的一个或多个软件应用。此外,处理单元可以响应于软件的执行来访问、存储、操控、处理和生成数据。本领域技术人员将理解,尽管为了便于理解,可以图示单个处理单元,但是该处理单元可以包括多个处理元件和/或多个类型的处理元件。例如,中央处理单元可以包括多个处理器或者一个处理器和一个控制器。此外,处理单元可以具有不同的处理配置,诸如并行处理器。
[0027]根据本发明构思的上述示例实施例的方法可以利用程序指令实现,该程序指令可以由计算机或处理器执行并且可以记录在计算机可读介质中。该介质还可以包括单独的或与程序指令相组合的数据文件、数据结构等。记录在介质中的程序指令可以特别针对本发明构思的示例实施例来设计和配置,或者对于计算机软件领域的技术人员来说是已知的和可用的。计算机可读介质包括磁介质,诸如硬盘、软盘和磁带;光学介质,诸如致密盘-只读存储器(CD-ROM)盘和数字多功能盘(DVD);磁光介质;以及被专门配置为存储和执行程序指令的硬件设备,诸如只读存储器(ROM)、随机存取存储器(RAM)、闪速存储器等。程序指令包括诸如由编译器产生的机器代码以及可以由计算机使用解释器执行的高级代码两者。所描述的硬件设备可以被配置为执行一个或多个软件模块,以执行本发明构思的上述示例实施例的操作,或者反之亦然。
[0028]然而,应当记住,所有这些和类似术语应与适当的物理量相关联,并且仅是应用于这些量的方便标签。除非另有特别声明或者如从讨论中清楚,否则诸如“处理”或“计算”或“运算”或“确定”或“显示”等术语指代计算机系统或类似的电子计算设备/硬件的动作和过程,其将表示为计算机系统的寄存器和存储器内的物理电子量的数据操控和变换成类似地表示为计算机系统存储器或寄存器或其他这样的信息存储、传输或显示设备内的物理量的其他数据。
[0029]应当理解,上面的描述中解释的实施例仅是说明性的,并且不限制本发明的范围。设想了许多这样的实施例和描述中解释的实施例的其他修改和改变。本发明的范围仅由权利要求书的范围来限定。