Usability improvements for visual interfaces

One or more aspects relate to a screen reader. Markup code for a visual interface containing at least one visual control is located. A screen reader menu for the visual interface is created. A visual control name and screen location, both corresponding to the at least one visual control, are extracted from the markup code. A menu item is generated in the screen reader menu corresponding to the at least one visual control. The menu item is set to simulate an action at the screen location corresponding to the at least one visual control on selection of the menu item whereby on user selection of the menu item the action is simulated at the screen location corresponding to the at least one visual control.

BACKGROUND

One or more aspects relate to visual interfaces. In particular, one or more aspects relate to improving the usability of a visual interface.

One or more aspects operate in the general environment of screen readers and applications with visual interfaces, such as a visual integrated development environment (visual IDE).

Users of screen readers typically have three ways of moving around a screen, the arrow keys, the tab key or special keystrokes which are either built into the screen reader or the application itself. It is a common experience for a screen reader user not to reach a part of an application screen comprising a visual control and also not to be able to activate a visual control in part of an application screen that a user can access. So users with no vision may not even know there are unreachable screen parts because a screen cursor cannot always reach or see every screen part. The net result is that screen reader users have limited access to applications and should not assume that they will be able to access every visual control on a new application.

An example application that uses visual controls that are not normally accessible to a screen reader is a visual IDE. Visually impaired users are unable to navigate a visual development canvas, and thus cannot readily access or generate visual controls (for example message flow controls). A solution is required to enable visually impaired users to access these visual controls and other graphical user interface (GUI) elements in order to create and modify applications (for example message flow applications).

SUMMARY

In a first aspect of the invention, there is provided a screen reader comprising a markup code probe to locate markup code for a visual interface containing at least one visual control; a user menu factory to create a screen reader user menu for the visual interface; a visual control extraction engine to extract properties from the at least one visual control, the properties including a visual control name and a visual control screen location, corresponding to the at least one visual control; a menu item factory to generate a menu item in the screen reader menu corresponding to the at least one visual control; and a menu controller to set the menu item to simulate an action at the screen location corresponding to the at least one visual control on selection of the menu item whereby on user selection of the menu item the action is simulated at the screen location corresponding to the at least one visual control.

In one embodiment, a user interface is created that is for a screen reader arrow key input but any other input could be used including a numbered menu item or programmable shortcut keys or non-keyboard input such as voice recognition.

In one embodiment, the visual control extraction engine is also for extracting visual control actions however, an embodiment can work by assuming that the visual control action is one or more of: a right mouse click; a left mouse click; a cursor hover or a visual control selection.

The embodiments describe analyzing the underlying markup code for a visual interface (for example extendable markup language (XML) code describes the message flow in a message flow integrated development environment) to get coordinates of at least one visual control (sometimes called a node) and provide an accessible menu (for example an arrow key menu) to simulate mouse clicks, mouse hovers or other user selection or user visual interface action. This enables a visually impaired user to achieve focus on a control node within the visual interface (for example a canvas) to expose a further menu or menus (for example panels) appropriate to that visual control (for example properties of a message flow input node).

One or more embodiments create appropriate accessible items, for instance an additional menu that is generated based on analysis of the markup code underlying the visual interface. The embodiments access the source of the visual interface using appropriate application programming interfaces (for example: “locate source” in an integrated development environment or a typical Internet browser).

The embodiments provide access for visual environments (for example visual integrated development environments) for impaired users and provide enhanced access to those applications for sighted users.

In a second aspect of the invention, there is provided a method for a screen reader comprising locating markup code for a visual interface containing at least one visual control; creating a screen reader menu for the visual interface; extracting from the markup code a visual control name and a screen location, corresponding to the at least one visual control; generating a menu item in the screen reader menu corresponding to the at least one visual control; and setting the menu item to simulate an action at the screen location corresponding to the at least one visual control on selection of the menu item whereby on user selection of the menu item the action is simulated at the screen location corresponding to the at least one visual control.

In one embodiment, the steps of extracting, generating and setting are performed for each visual control in the markup code for the visual interface.

Further, the method further includes locating a dependent visual control depending from the at least one visual control and extracting from the markup code a dependent visual control name and screen location, corresponding to the at least one visual control.

In one or more aspects, the method further includes locating a doubly dependent visual control depending from the at least one dependent visual control and extracting from the markup code a doubly dependent visual control name and screen location, all corresponding to the at least one visual control.

Yet further, the method includes extracting from the markup code further descriptive information relating to the visual control and making such further descriptive information available from the menu item.

In a third aspect of the invention, there is provided a computer program product for a screen reader, the computer program product comprising a computer readable storage medium having program instructions embodied thereon, the program instructions executable by a processor to cause the processor to locate markup code for a visual interface containing at least one visual control; create a screen reader menu for the visual interface; extract from the markup code, a visual control name and screen location, both corresponding to the at least one visual control; generate a menu item in the screen reader menu corresponding to the at least one visual control; and set the menu item to simulate an action at the screen location corresponding to the at least one visual control on selection of the menu item whereby on user selection of the menu item the action is simulated at the screen location corresponding to the at least one visual control.

The computer program product comprises a series of computer-readable instructions either fixed on a tangible medium, such as a computer readable medium, for example, optical disk, magnetic disk, solid-state drive or transmittable to a computer system, using a modem or other interface device, over either a tangible medium, including but not limited to optical or analog communications lines, or intangibly using wireless techniques, including but not limited to microwave, infrared or other transmission techniques. The series of computer readable instructions embodies all or part of the functionality previously described.

DETAILED DESCRIPTION

Referring toFIG. 1, the deployment of a one embodiment in a computer processing system10is described. Computer processing system10is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing processing systems, environments, and/or configurations that may be suitable for use with computer processing system10include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed computing environments that include any of the above systems or devices. A distributed computer environment may include a cloud computing environment for example where a computer processing system is a third party service performed by one or more of a plurality of computer processing systems. A distributed computer environment may also include an Internet of Things computing environment for example where computer processing systems are distributed as a network of objects that can interact with a computing service.

Computer processing system10may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer processor. Generally, program modules may include routines, programs, objects, components, logic, and data structures that perform particular tasks or implement particular abstract data types. Computer processing system10may be embodied in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

Computer processing system10includes a general-purpose computer server12and one or more input devices14and output devices16directly attached to the computer server12. Computer processing system10is connected to a network20. Computer processing system10communicates with a user18using input devices14and output devices16. Input devices14include one or more of a keyboard, a scanner, a mouse, trackball or another pointing device. Output devices16include one or more of a display or a printer. Computer processing system10communicates with network devices (not shown) over network20. Network20can be a local area network (LAN), a wide area network (WAN), or the Internet.

Computer server12includes a central processing unit (CPU)22; a network adapter24; a device adapter26; a bus28and memory30.

CPU22loads machine instructions from memory30and performs machine operations in response to the instructions. Such machine operations include, for instance, incrementing or decrementing a value in a register; transferring a value from memory30to a register or vice versa; branching to a different location in memory if a condition is true or false (also known as a conditional branch instruction); and adding or subtracting the values in two different registers and loading the result in another register. A typical CPU can perform many different machine operations. A set of machine instructions is called a machine code program, and the machine instructions are written in a machine code language which is referred to a low level language. A computer program written in a high level language needs to be compiled to a machine code program before it can be run. Alternatively a machine code program such as a virtual machine or an interpreter can interpret a high level language in terms of machine operations.

Network adapter24is connected to bus28and network20for enabling communication between the computer server12and network devices.

Device adapter26is connected to bus28and input devices14and output devices16for enabling communication between computer server12and input devices14and output devices16.

Memory30includes computer system readable media in the form of volatile memory32and non-volatile or persistent memory34. Examples of volatile memory32are random access memory (RAM)36and cache memory38. Examples of persistent memory34are read only memory (ROM) and erasable programmable read only memory (EPROM). Generally volatile memory is used because it is faster and generally non-volatile memory is used because it will hold the data for longer. Computer processing system10may further include other removable and/or non-removable, volatile and/or non-volatile computer system storage media. By way of example only, persistent memory34can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically a magnetic hard disk or solid-state drive). Although not shown, further storage media may be provided including: an external port for removable, non-volatile solid-state memory; and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a compact disk (CD), digital video disk (DVD) or Blu-ray. In such instances, each can be connected to bus28by one or more data media interfaces. As will be further depicted and described below, memory30includes a program product having a set of program modules that are configured to carry out the functions of embodiments of the invention.

The set of program modules includes, for instance, visual editor module100; visual application102; screen reader module104; menu module106; and visual reader module200. In one embodiment, ROM in memory30stores the modules that enables the computer server12to function as a special purpose computer specific to the modules. Further program modules that support one or more embodiments but are not shown include, for instance, firmware, boot strap program, operating system, and support applications. Each of the operating system, support applications, other program modules, and program data or some combination thereof, may include an implementation of a networking environment.

Computer processing system10communicates with at least one network20(such as a local area network (LAN), a general wide area network (WAN), and/or a public network like the Internet) via network adapter24. Network adapter24communicates with the other components of computer server12via bus28. It should be understood that although not shown, other hardware and/or software components could be used in conjunction with computer processing system10. Examples, include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, redundant array of independent disks (RAID), tape drives, and data archival storage systems.

Visual editor module100is typically a visual integrated development environment whereby an application can be edited in a text mode or in a graphical user interface.

Visual application102is a program that is edited by the visual editor module100. Primarily the application is a coded application in a language such as hypertext markup language or extensible markup language.

Screen reader module104has typical screen reader functionality for accessible parts of a computer screen except that it interacts with a visual reader module of one or more embodiments to reach inaccessible parts of an application screen (in this example the visual application102as edited by visual editor module100).

Menu module106is a resultant data file of one or more embodiments used by the screen reader module104that describes for the screen reader module104the created reader elements corresponding to the visual application102so that the screen reader module104reads the created reader elements as a menu of items. In one embodiment, an arrow menu user interface is created for screen reader arrow key input, but any other type of input could be used, including a numbered menu item or programmable shortcut keys or non-keyboard input, such as voice recognition.

Visual reader module200comprises functionality of one or more embodiments and is described in more detail below.

Referring toFIG. 2, visual reader module200comprises, for instance, the following components: markup code probe202; menu factory204; visual control extractor engine206; menu item factory208; menu controller210; and visual reader method300.

Markup code probe202is for locating markup code for a visual interface containing at least one visual control. In one embodiment, markup code probe202is further for locating at least one dependent visual control depending from the at least one visual control. In one embodiment, markup code probe202is further for locating at least one doubly dependent visual control depending from the at least one dependent visual control, and the visual control extraction engine is further for extracting from the markup code, a dependent visual control name and screen location, both corresponding to the at least one doubly dependent visual control.

Menu factory204is for creating a screen reader menu for the visual interface.

Visual control extraction engine206is for extracting from the markup code: visual control name and screen location, all corresponding to the at least one visual control. In one embodiment, visual control extractor engine206is further for performing the steps of extracting, generating and setting for each visual control in the markup code for the visual interface. Also, in one embodiment, visual control extractor engine is further for extracting from the markup code further descriptive information relating to the visual control and making such further descriptive information available from the arrow menu item. For instance, the visual control may have a visual control label; and a visual control hover description. It is envisaged that the visual control extractor engine is also for extracting visual control actions; however, one embodiment works by assuming that the visual control action is one or more of a right mouse click; a left mouse click; a cursor hover or a visual control selection.

Menu item factory208is for generating a menu item in the menu corresponding to the at least one visual control.

Menu controller210is for setting the menu item to simulate the action at the screen location corresponding to the at least one visual control on selection of the menu item.

Visual reader method300is for controlling the above components according to a method300A of one embodiment or method300B of another embodiment, both described in more detail below.

Referring toFIG. 3A, visual reader method300comprises visual reader method300A in one embodiment including logical process steps302A to318A.

Step302A is the start of the method when screen reader application requests source information for any graphical user interfaces on the screen.

Step304A is for locating markup code or source information for a graphical user interface containing at least one visual control by calling an application program interface for the visual editor or for the operating system. The markup code is then searched for a first visual control.

Step306A is for creating a screen reader menu for the visual application.

Step308A is for defining a loop for all the visual controls in the visual application.

Step310A is for extracting from the markup code visual descriptions comprising: visual control name and screen location, all corresponding to the at least one visual control. In one embodiment, the markup code visual descriptions further comprise: a visual control label; and a visual control hover description. One embodiment assumes that the control action is one of: a left mouse click, a right mouse click; a hover; or a selection. However, it is envisaged that the control action could also be extracted from the markup code.

Step312A is for generating a menu item in the screen reader menu corresponding to the at least one visual control.

Step314A is for setting the menu item to simulate the action at the screen location corresponding to the at least one visual control on selection of the menu item.

Step316A is for looping back to step308A for each visual control in the visual application. In one embodiment, each defined visual control comprises all dependent visual controls and all further dependent visual controls when there is a hierarchy of dependent visual controls.

Step318A is the end of visual reader method300A.

Referring toFIG. 3B, visual reader method300comprises visual reader method300B in an alternative embodiment including logical process steps302B to318B.

Step302B is the start of the method when screen reader application requests source information for any graphical user interfaces on the screen.

Step304B is for calling an operating system application programming interface (API) to find the visual application and a first visual control in the visual application.

Step306B is for creating a screen reader menu for the visual application.

Step308B is for defining a loop for all the visual controls in the visual application.

Step310B is for calling an application programming interface (API) in relation to the located visual control to locate visual descriptions comprising: visual control name and screen location, all corresponding to the at least one visual control. In one embodiment, the markup code visual descriptions further comprise: a visual control label; and a visual control hover description. In a variation, the visual control action (for example selecting the control with a left or right mouse click or hovering over mouse) may be determined by calling an API.

Step312B is for generating a menu item in the screen reader menu corresponding to the at least one visual control.

Step314B is for setting the menu item to simulate the action at the screen location corresponding to the at least one visual control on selection of the menu item.

Step316B is for looping back to step308B for each visual control in the visual application. In one embodiment, each defined visual control comprises all dependent visual controls and all further dependent visual controls when there is a hierarchy of dependent visual controls.

Step318B is the end of visual reader method300B.

Referring toFIG. 4, an example markup code visual application is shown in extensible markup language (XML). The graphical representation of the markup code is shown inFIG. 5Aand an arrow menu representation is shown inFIG. 5B.

The first line of the code states that version 1.0 of XML is used throughout.

The second line defines an “Application” element which has a child “composition” element. The “composition” element in turn has 3 child elements: node elements; connections elements; and graphic elements.

Two node elements are defined in the code, one for each node on the canvas. Each node element has attributes for the node type (“type”), a display name (“displayName”), the X,Y coordinates of the top left corner of the associated graphic on the canvas (“location”) and some additional properties associated with the node, in this case “queueName”, “connection”, “destinationQueueManagerName” and “transactionMode”. The runtime environment is for instantiating a node element.

The first node element is shown in underlined bold and reproduced below:<nodes type=“ComIbmMQInput.msgnode”_displayName=“MQInputNode”id=“FCMComposite_1_1”_location=“89,97”queueName=“CSIM_SERVER_IN_Q”_connection=“SERVER”destinationQueueManagerName=“CSIM”_transactionMode=“no”/>

The tag “nodes type=” defines an input control (generally called nodes in this example application) called “FCMComposite_1_1” at location coordinates 89, 97. The node type that will be instantiated is a definition held by the run time environment called “ComIbmMQInput.msgnode”.

Second control defining code lines are shown in bold and reproduced below:<nodes type=“ComIbmMQOutput.msgnode”displayName=“MQOutputNode”id=“FCMComposite_1_2”_location=“258,97”queueName=“CSIM_COMMON_REPLY_Q”_connection=“SERVER”destinationQueueManagerName=“CSIM”_transactionMode=“no”persistenceMode=“no”/>

The tag “nodes type=” defines an output control called “FCMComposite_1_2” at location 258, 97.

There is only one connection here in that the first node is connected to the second node. The connection has a “type” and an “id”, the attributes then further define that the connection is between the “OutTerminal.out” terminal (“sourceTerminalName”) of a first node with id ‘FCMComposite_1_1’ (“sourceNode”) and the “InTerminal.in” terminal (“targetTerminalName”) of a second node with id ‘FCMComposite_1_2’ (“targetNode”)—thus a connection from the MQInputNode to the MQOutputNode.<connections_type=“FCMConnection”_id=“FCMConnection_1”targetNode=“FCMComposite_1_2”_sourceNode=“FCMComposite_1_1”sourceTerminalName=“OutTerminal.out”targetTerminalName=“InTerminal.in”/>

Graphic elements define the graphic icons associated with each of the nodes by type. Both are 32×32 pixels (“width” and “height” attributes) with the “resource” attribute defining the file containing the associated graphic for a node with a type as defined by the “type” attribute (thus can be cross-referenced to the “type” attribute for the “nodes” elements).<colorGraphic_type=“ComIbmMQInput.msgnode”_resource=“icon1.gif”width=“32”_height=“32”/><colorGraphic_type=“ComIbmMQOutput.msgnode”_resource=“icon2.gif”width=“32”_height=“32”/>

Referring toFIG. 5A, there is shown an example screen showing a state of a graphical user interface (GUI)500of one embodiment. GUI500comprises: window control502; toolbar504; and frame506.

Window control502provides for minimizing, maximizing and closing of GUI500.

Save504.2is a control for saving input data in a particular state.

Load504.4is a control for loading prompt and user data. Saving and loading of prompt and user data.

Setting504.6provides a user control to change a setting for opening GUI500.

Operating system controls, like502and504, are normally accessible to a screen reader because they are operating system based. However, in our example frame506is not accessible because it is a proprietary application and not operating system based.

Frame506is for displaying the graphical user interface of a visual application that is inaccessible to a normal screen reader because it is application based. Frame506comprises input visual control508.1(represented by an input icon) and output visual control508.2(represented by an output icon). Both visual controls are normally inaccessible from a screen reader but the embodiments access the underlying code of the visual application and create an arrow menu for the screen reader.

Referring toFIG. 5B, an example menu created by the embodiments is shown. Menu510is created with two menu items: FCMComposite 1 1 and FCMComposite 1 2. FCMComposite 1 1 corresponds to visual control508.1and performs a mouse click substantially at location 89, 97 when selected (@Mouse Click at location 89, 97). FCMComposite 1 2 corresponds to visual control508.2and performs a mouse click substantially at location 258, 97 when selected (@Mouse Click at location 258, 97).

The coordinates 89, 97 refer to an x, y coordinate for the top left position of the icon for the first component described in the XML. The coordinates 258, 97 refer to an x, y coordinate for the top left position of the icon for the second component described in the XML.

The mouse-click (in this case) should not be performed at this location specifically, but should be performed in the center of the icon graphic: mouse click at position where x coordinate=X+(½*width of icon graphic) and y coordinate=Y+(½*height of icon graphic).

Thus, for the modified example previously, this would be:

The simulated mouse clicks select the associated visual control and as such the IDE panels will update to display the property panels for the node which the user can access using existing keyboard actions to navigate regular GUI elements.

Further embodiments of the invention are now described. It will be clear to one of ordinary skill in the art that all or part of the logical process steps of one or more embodiments may be alternatively embodied in a logic apparatus, or a plurality of logic apparatus, comprising logic elements arranged to perform the logical process steps of the method and that such logic elements may comprise hardware components, firmware components or a combination thereof.

In a further alternative embodiment, one or more aspects of the present invention may be realized in the form of a computer implemented method of deploying a service comprising steps of deploying computer program code operable to, when deployed into a computer infrastructure and executed thereon, cause the computer system to perform all the steps of the method.

It will be appreciated that the method and components of one or more embodiments may alternatively be embodied fully or partially in a parallel computing system comprising two or more processors for executing parallel software.

A further embodiment of the invention is a computer program product defined in terms of a system and method. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to carry out aspects of the present invention.

It will be clear to one skilled in the art that many improvements and modifications can be made to the foregoing exemplary embodiment without departing from the scope of the present invention.