Patent Publication Number: US-10769183-B2

Title: Identifying resources in user interfaces for feedback

Description:
RELATED APPLICATION 
     This application claims an invention which was disclosed in Provisional Application No. 62/502,583, filed May 5, 2017, entitled “IDENTIFYING RESOURCES IN USER INTERFACES FOR FEEDBACK”. The benefit under 35 USC § 119(e) of the United States provisional application is hereby claimed, and the aforementioned application is hereby incorporated herein by reference. 
    
    
     BACKGROUND 
     Complex software systems often include many source code resources that, through compilation, linking, and/or runtime actions, are loaded into memory and displayed in user interfaces. Typically, resources in a software project are isolated into resource files where resources are included and referenced or indexed by resource identifiers. The source code files in the software project include resources by referencing the necessary resource identifiers. For example, most user interface buttons have a label property. Source code might assign a resource identifier to a label of an instance of the button. The resource identifier may point to a localized resource string, e.g., “hello world”. When the source code is compiled and executed, the button is displayed with the “hello world” label. Using this layer of indirection for resources, it is convenient to change the locale (or language) for the resources for a software project. Usually, the resource strings are resolved at runtime. For instance, if an environment variable specifies a locality or language, that variable is used at runtime to select the resource file that will be used to resolve the resource strings. 
     Large software projects such as operating systems or complex web applications may have thousands or even millions of resource strings. Maintaining so many resource strings, for instance correcting errors or updating for consistency, is burdensome. If a software project is to be provided for different languages or locales, each different locale or language requires its own set of resource string translations or versions. Consider that there are over 7,000 languages spoken in the world. Roughly 150 to 200 of the world&#39;s languages are spoken by at least one million people. Yet even well-funded software projects with many decades of development often support no more than a hundred languages or dialects. As can be seen from these numbers, most software projects lack resource support for languages spoken by significant numbers of people. The practical inaccessibility of many software products means that much potential human productivity is unrealized. 
     One approach to this problem has been to enable communities of users to build or revise the resources of projects. With this approach, a feedback system is designed to allow users to provide feedback in association with a resource. Typically, workflows for such feedback have been limited to basic functions such as receiving a feedback from a user (e.g., a proposed edit or translation) and associating the feedback with the relevant software project or a generic screen or topic of the relevant software. Developers, translators, or editors then review the feedback and make appropriate edits or additions to the resources of the software project. 
     The current approach for managing feedback has many shortcomings. To make use of a feedback, a person (reviewer) must go through several steps. The feedback itself must be reviewed for accuracy and relevance. To review the feedback, and to apply it to a resource file, the reviewer must understand the context of the resource to which the feedback is directed. This may involve manual steps of tracking down the resource in source code, running the relevant software, using available clues to locate the resource. This alone can take significant time, even if the relevant user interface screen or application has been identified. Review of feedback such as a proposed resource addition or edit may involve verifying a proposed language translation or evaluating the propriety of a proposed change. This human involvement is costly and can slow down the time that it takes for feedback to be incorporated into a software project. 
     Techniques related to facilitating feedback for resource strings are discussed below. 
     SUMMARY 
     The following summary is included only to introduce some concepts discussed in the Detailed Description below. This summary is not comprehensive and is not intended to delineate the scope of the claimed subject matter, which is set forth by the claims presented at the end. 
     Embodiments relate to enabling users to provide feedback on resources displayed in user interfaces when the source-level identity of the resources is unknown during runtime. A feedback tool is executed to allow a user to specify any arbitrary resource displayed in a user interface. When the feedback tool is activated, context metadata about the user interface is captured and stored. The context metadata corresponds to runtime activity such as loading executable modules, rendering the user interface, handling windowing/input events, and the like. The feedback tool is independent of the user interface and is used to specify a resource displayed in the user interface. Specifying or selecting a resource for feedback triggers capture of the specified resource and the relevant runtime context metadata. The resource and context metadata are used as query parameters to query a database of known resources and their respective properties and identifiers. If a match is found, then the source-level identity of the resource is known. The feedback is then associated with the resource and can be used to directly modify, replace, or add to the resource (for instance in a new language/locale). A community of users using the feedback tool may collectively maintain resource files or create new resource files for new language support. 
     Many of the attendant features will be explained below with reference to the following detailed description considered in connection with the accompanying drawings. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present description will be better understood from the following detailed description read in light of the accompanying drawings, wherein like reference numerals are used to designate like parts in the accompanying description. 
         FIG. 1  shows how a user interface can display a resource without having information sufficient to identify the resource with the particularity needed for updating the resource. 
         FIG. 2  shows how a feedback tool can be used to enable pinpoint identification of resources displayed in user interfaces. 
         FIG. 3  shows details of an inference engine. 
         FIG. 4  shows a network-based feedback system for enabling communities of participants to provide feedback via a network to software projects. 
         FIG. 5  shows a process for providing resource feedback. 
         FIG. 6  shows details of a computing device on which embodiments described above may be implemented. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  shows how a user interface  100  can display a resource  102  without having information sufficient to identify the resource  102  with the particularity needed for updating the resource  102 . The user interface  100  may be any type of graphical interface displayed by any type of software on any type of computing device. In one embodiment, the user interface  100  may be part of a stand-alone compiled application consisting of binary executable files  104 , including perhaps dynamically linked libraries, directly executable files, etc. The executable files  104  are compiled from source files  106 . The executable files  104  may be pre-compiled machine code, just-in-time compiled bytecode, or the like. The source files  106  may include programming language source code as well as resource files  108 . As noted in the Background, a resource file  108  may associate resources with respective resource identifiers (resource IDs)  110 . A source code file has source code instantiating the user interface  100  and referencing the resource ID  110  to assign the corresponding resource  102  to a user interface element  112 . When the source code is compiled, the resource ID is embedded in an executable file  104 . When the executable file is executed, the runtime environment reads the resource ID, accesses the corresponding resource file  108 , obtains the resource  102  per the resource ID, and displays the resource  102  in the user interface element  112 . 
     This simple example illustrates the general approach of using an indirection layer to allow resources to be conveniently incorporated into a software project. The details may vary with different programming languages and different runtime environments. For example, some compilers may embed referenced resources directly into compiled executable files  104 . Some may have special sub-systems for building and executing user interfaces. In most cases, there may be a complex chain of files and dependencies when a resource is referenced. Furthermore, in most cases it is difficult or impossible to tell, at runtime, where any particular resource may have been defined. That is, during execution of the software that displays the user interface  100 , there is usually no information available that directly relates the resource  102  as displayed in the user interface element  112  to its definition or provenance in the resource file. Thus, if feedback about the resource or a new translation of the resource is submitted, additional instrumentation is needed to enable a resource identified by a user to be correlated, at the source code level, with feedback from the user. 
       FIG. 2  shows how a feedback tool  130  can be used to enable pinpoint identification of resources displayed in user interfaces. A runtime environment  132  executes binary executables  104 . The runtime environment  132  may be a managed environment such as a Java™ virtual machine, an operating system (with a loader, linker, etc.), or any other known system for managing the loading and execution of compiled code. The runtime environment executes an application that displays the user interface  100 . Either before the application starts executing, or during execution of the application, the feedback tool  130  is started. In one embodiment, the feedback tool  130  is a distinct application started manually by the user  134 . The feedback tool  130  is designed to be able to work with any arbitrary code or application that executes on the computing device and displays a user interface. The feedback tool  130  need not have any information about any resource strings that will be displayed or potentially selected for feedback. 
     Conceptually, the feedback tool  130  may be thought of as interactive instrumentation layered on top of the user interface  100 . In one embodiment, the feedback tool  130  is part of the event-handling chain and may intercept input events, handle them, and also allow the events to pass on to the underlying user interface. The feedback tool  130  may have a toolbar, pop-up menu, or other controls to enable the user  134  to control the behavior of the feedback tool  130 . For example, start/stop buttons may activate or deactivate the feedback tool  130 . While active, the feedback tool  130  includes a mechanism to enable the user to manually select or target a displayed resource. While the feedback tool  130  is active, a resource may be targeted by directing a pointer input to a resource, by hovering a pointer over a resource, clicking the resource, activating a control of the feedback tool  130 , etc. Interactive selection of a resource displayed while the feedback tool  130  is active also triggers display of a feedback input control (e.g., a text control) into which user feedback  136  may be inputted. In one embodiment, properties of the feedback may also be inputted. For example, if the feedback is a proposed translation of a resource string, the feedback tool  130  may include an interactive control to set the language of the feedback. 
     The feedback tool  130  also captures context metadata  138 . As discussed above, in many runtime environments, resources are loaded during execution without the ready availability of the source-level references used to compile the code that uses resources. Therefore, to enable identification of a resource in terms of its source code (or resource file) and hence revision, augmentation, or extension of the resource, context metadata is captured during execution and then used to infer the identity of the resource. 
     The captured context metadata may include some of the following types of information. The process name of the process displaying the resource may be captured. A current culture environment variable may also be read and stored. In some cases, user interface instrumentation may be available to capture UI automation identifiers or the like. Bounding boxes and screen coordinates may also be stored. Some systems may even enable full reconstruction of the hierarchy of user interface elements in the target user interface. Resource paths, if supported, may be acquired. Names and locations of files loaded in association with the target application or user interface may be captured. File dependencies may be included. In some embodiments, context capture is in a constant-on state where context is continually captured when the tool is running, for instance in a circular buffer, and is persisted when there is an indication that feedback is to be provided. A list of modules loaded, assembly version, and the like may also be stored. Configuration settings, registry settings, and other system data is also potentially helpful for inferring a resource&#39;s underlying identity. Some runtime environments and/or compiled code may include features designed for accessibility which can also be leveraged. Significantly, the context metadata  138  also includes the actual resource targeted for feedback. Also, a screenshot of the display or user interface is captured and stored. 
     At least some of the context metadata  138  is fed to an inference engine  140  (when the feedback is captured, or later). As will be described with reference to  FIG. 3 , the inference engine accesses a resource database  142  containing pre-accumulated static information about resources used for the relevant software subject to feedback. Preferably, the resource database  142  contains as many of the same pieces of metadata as potentially captured by the feedback tool  130  into the context metadata  138 . Briefly, the inference engine  140  performs a query of the resource database  142  to find any resources matching the captured resource and also having associated resource metadata that matches the captured context metadata. Ideally, this will identify a single resource. The resource database  142  also includes a universal resource identifier (URI) for each known resource. The URIs form a namespace within which all potential resources (including for different applications/executables) have respective unique identifiers. The resource database  142  may also have the resource IDs used in the relevant source code projects to identify resources. Put another way, the resource database  142  includes: information for identifying resources from context information, globally unique identifiers for resources that may be used by the feedback tool  130  (and by other components described later), as well as project or source code resource IDs that allow a resource to be linked to the source code file or resource file in which the resource is defined and/or referenced. 
     If the inference engine  140  finds a match for the targeted resource, the URI  144  (or a sufficiently qualified resource ID) is outputted and stored in a feedback table  146 . The feedback table  146  stores the URIs  148  of resources targeted for feedback in a one-to-many relationship with each item of feedback, feedback properties (e.g., language), the screenshot (if taken) and any other information that might be relevant to feedback directed to resources. The feedback table  148  may also store references to persisted context metadata for feedback for which the inference engine  140  is unable to resolve unique resource identifiers. 
     The resource database  142  and the feedback table  148  may be functionally linked to a code database or code repository  150 . The code repository may store the source code files  106 , including resource files  108 . Code repository  150  may store source and resource files for any of the software potentially targeted by the feedback tool  130 . The code repository  150  (or the feedback table  146 ) may store a map between URIs and the resource IDs in resource files so that global URIs may be mapped to specific resource IDs in specific resource files. In one embodiment, URIs may be implemented as a concatenation of the code-level resource ID, plus a file name/path, and/or a project name, or other attributes that together can globally uniquely identify resources. 
     The code repository  150  is also functionally linked to the resource database  142 . Specifically, the resource database  142  is populated with resources and respectively corresponding source metadata from the code repository  150 . That is, properties of the resources are obtained from the source code files, project files, revision control system, and similar sources, and stored in association with the resources. As shown in  FIG. 2 , there is a feedback loop from the resource files in the code repository, to the binary executables  104  running the user interface  100  displaying resources, to the feedback tool  130 , and back to the resource files in the code repository  150 . 
       FIG. 3  shows additional details of the inference engine  140 . As discussed above, the context metadata  138  is captured by the feedback tool during execution of a target user interface. The feedback tool, based on user input, captured runtime data  158  during loading and execution of compiled code. For instance, file names/paths, module names, revision numbers, language/locale settings, windowing events, and other datum mentioned above. In some embodiments, the feedback tool and/or the inference engine  140  may construct a model of the user interface, building a representation of the objects (and their properties) of the target user interface, the screen geometry of the user interface, and the like. The inference engine  140  performs a process  162  to attempt to resolve the captured resource  160  and the captured runtime data  158 . 
     The inference engine  140  receives the captured resource and runtime context data, and optionally any model or reconstruction of the user interface. The inference engine  140  builds a query specifying the captured resource and accompanying properties or context that are determined to be specific to the resource. The inference engine  140  then performs the query against the resource database  142 . If a single resource matches the query then the resource is presumed to be the same resource queried for. An identifier of the resource is outputted, including perhaps a URI (which may be generated for the resource if one does not already exist), is outputted. The output of the inference engine  140  may go to the feedback tool, which in turn stores the feedback, resource identity, etc. In another embodiment, the inference engine  140  stores its output to the feedback table  146 . 
     The embodiments described above may be performed on a single computing device storing the feedback tool, the executable components, the inference engine, the resource database, and the feedback table. The locally-implemented embodiments may also store resource files. The feedback inputting, resource identifying, and the storing of feedback in association with resources is executed in the context of the local operating system. The feedback may even be incorporated into new or existing resource files and hence become available locally in the binary executables and the corresponding user interfaces. Even if the user or the runtime environment does not know the precise identify of feedback-targeted resources, the resources may be identified and supplemented, replaced, added (perhaps for different localizations or languages), and so forth. As described next, network-based embodiments may enable feedback to be provided and managed in a communal fashion, which may provide various other advantages. 
       FIG. 4  shows a network-based feedback system for enabling communities of participants  170  to provide feedback via a network  172  to software projects. The components involved are functionally similar to those described above. Feedback inputting and resource context capturing is performed on local devices operated by the users, but the feedback is passed back through the network  172  to a feedback hub  174 , which may be a website, a network service, or similar network-based application-level software executing on one or more server devices. 
     The feedback hub  174  may interface with a community management module  176  that manages user identities for the participants  170 . The community management module  176  may manage login credentials (or using third-party credentials), maintain authorization levels of the user identities, store user profiles, manage user groups, and the like. The feedback hub  174  receives feedback data from the participant devices, for example, copies of the local feedback tables, but at the least, indicia of feedback-targeted resources, the feedback itself, etc. In one embodiment resource context is passed back to the feedback hub  174  and the feedback hub  174  runs the inference engine to resolve the resource context metadata to specific source-level resources. 
     In some embodiments, user authorization levels may be used to determine how feedback from participants is handled. For example, low-authorization users may be permitted to submit feedback and the feedback is stored for later review and perhaps approval/denial by privileged users/reviewers. Medium-authorization users may be permitted to submit feedback that is applied to resource files under limited conditions (e.g., for resources in languages that the users are approved for). High-authorization users may be permitted to submit feedback for any resource and the feedback is applied to the relevant source code. Different users may have different authorization levels within different user groups. In one embodiment all users are permitted to make local resource changes that are used only locally, and their feedback (resource changes) are forwarded for collection and possible community incorporation. 
     As can be seen, this approach enables the community to control the resource content of the relevant distributed executable user interfaces being locally executed. A feedback collection module  178  collects feedback from users and groups the feedback by software project or application, localization (language), and so forth. The community may even be permitted to initiate new resource files for new languages or locales. Thus, in cases of “orphan” languages with too few speakers to justify paid large-scale translations of complex resource sets, the community of speakers of that language may collectively build, maintain, and use their own localized resources. If the executables load resources at runtime per a localization setting (i.e., if resources are not copied into compiler outputs), then the relevant executables need not be modified. The community may continue to use the same software and begin using their new resources with re-building the software. The publisher or developer of the software need not do any more than provide or subscribe to supporting infrastructure to enable use of the feedback tool and the feedback hub. 
     A hybrid local-community design may also be used by combining the local and community elements. The feedback tool and target software execute on a local device, capture feedback and context, locally infer resource identities, and locally update or supplement resource files. The local device also has a community component that sends the locally accumulated feedback, context metadata, etc. to a backend service. The backend service collects the feedback and metadata from many such user devices and may combine them into communal updates as per the community-based embodiment described above. 
     The resource ID metadata captured at runtime can be leveraged to provide translation recommendations for the same string resource provided by other members within a user community. The users are presented with these recommendations in a language-community application which includes a user interface enabling the users to vote on their preferred recommendations. An elected recommendation can be automatically rolled into the appropriate resource files. The language-community application can also serve as a channel to provide a response to an end user who submitted a feedback. The resource ID mapping system provides a means to track the state of the user&#39;s feedback all the way through a resolution of the feedback. The resource ID associated with each feedback item is used to query the feedback processing infrastructure and inform users about the review and implementation state of their feedback. The status of their feedback is provided back to the user through the language community application. The language-community application and the feedback tool  130  may be different screens/tabs of a same program, and the program may be web-based or locally installed. 
       FIG. 5  shows a process for providing resource feedback. The process may involve any combination of local and/or central actions as described above. As step  190  a user begins executing a user interface on their local device. The user interface display resources such as strings (note that other forms of resources besides text may be used; images, audio, and/or video resources may be rendered by the user interface). At step  192 , user input instructs the feedback tool that feedback is to be inputted. The feedback tool captures context data that may be useful for identifying the sources of the displayed resources. At step  194  user input is directed to a resource to identify the resource as a feedback target. Because the feedback tool is active, the user input is passed to the feedback tool rather than the user interface. At step  196  the feedback tool uses the input to identify the targeted resource and to store context metadata in association with a copy of the targeted resource. At step  198  the user uses the feedback tool to input feedback in association with the targeted resource, for instance, typing a new or updated translation, providing a comment, or other text. At step  200  the inference engine attempts to find the identity of the resource in its store of known resources (built from or comprised of resource files). Some of the context metadata may also be stored with the feedback (e.g., a screenshot of the user interface). At step  202 , if the target resource is inferentially resolved/identified, then the feedback is stored in association with the source-level identity of the resource. If the target resource is not identified then the feedback is stored in association with some or all of the captured context metadata and is available for human review. 
     In one embodiment, feedback that has been submitted for successfully identified resources is posted on the feedback hub for community review and may be approved or disapproved by a vote or a sufficiently privileged participant. 
     If the user interface is executed from declarative code then the declarative code may be analyzed to help reconstruct the original resource string. That is, if a resource string has been formatted, combined with other strings, and so forth, the string formatting code in the declarative code may be analyzed to help map the displayed text to the original resource string. 
       FIG. 6  shows details of a computing device  300  on which embodiments described above may be implemented. The technical disclosures herein will suffice for programmers to write software, and/or configure reconfigurable processing hardware (e.g., field-programmable gate arrays (FPGAs)), and/or design application-specific integrated circuits (ASICs), etc., to run on the computing device  300  to implement any of the features or embodiments described herein. 
     The computing device  300  may have one or more displays  350 , a network interface  354  (or several), as well as storage hardware  356  and processing hardware  358 , which may be a combination of any one or more: central processing units, graphics processing units, analog-to-digital converters, bus chips, FPGAs, ASICs, Application-specific Standard Products (ASSPs), or Complex Programmable Logic Devices (CPLDs), etc. The storage hardware  356  may be any combination of magnetic storage, static memory, volatile memory, non-volatile memory, optically or magnetically readable matter, etc. The meaning of the terms “storage” and “storage hardware”, as used herein does not refer to signals or energy per se, but rather refers to physical apparatuses and states of matter. The hardware elements of the computing device  300  may cooperate in ways well understood in the art of machine computing. In addition, input devices may be integrated with or in communication with the computing device  300 . The computing device  300  may have any form-factor or may be used in any type of encompassing device. The computing device  300  may be in the form of a handheld device such as a smartphone, a tablet computer, a gaming device, a server, a rack-mounted or backplaned computer-on-a-board, a system-on-a-chip, or others. 
     Embodiments and features discussed above can be realized in the form of information stored in volatile or non-volatile computer or device readable storage hardware. This is deemed to include at least hardware such as optical storage (e.g., compact-disk read-only memory (CD-ROM)), magnetic media, flash read-only memory (ROM), or any means of storing digital information in to be readily available for the processing hardware  358 . The stored information can be in the form of machine executable instructions (e.g., compiled executable binary code), source code, bytecode, or any other information that can be used to enable or configure computing devices to perform the various embodiments discussed above. This is also considered to include at least volatile memory such as random-access memory (RAM) and/or virtual memory storing information such as central processing unit (CPU) instructions during execution of a program carrying out an embodiment, as well as non-volatile media storing information that allows a program or executable to be loaded and executed. The embodiments and features can be performed on any type of computing device, including portable devices, workstations, servers, mobile wireless devices, and so on.