Methods and systems for determining screen-reader use

Methods, systems, and/or devices for determining that a user uses an assistive-technology program (such as a screen reader) for interacting with a web-based service are described. In some embodiments, this includes identifying a trigger event transmitted from a browser on a client device and identifying an inconsistency in the trigger event that correlates with use of an assistive-technology program in conjunction with the browser on the client device. In response to identifying the inconsistency, the method, system, or device may flag a user of the client device as a user of the assistive-technology program; and in response to flagging the user of the client device as a user of the assistive-technology program, provide content that is compatible with the assistive-technology program to the client device.

TECHNICAL FIELD

The disclosed implementations relate generally to determining whether a user of a computer is using screen reading technology.

BACKGROUND

Most people generally interact with computers through the use of a keyboard and/or a mouse to input information into the computer and through a display screen to receive information from the computer. However, people with blindness or severe sight impairment are often unable to use a display screen to interact with a computer. Accordingly, many such people rely on assistive-technology programs such as screen readers to receive information from computer systems.

A screen reading assistive-technology program reads the content on a screen aloud, so that it can be heard by a person who is unable to view the screen. Such programs are not limited to use by sight-impaired people, but the majority of users of screen reading programs have some level of sight impairment. There are multiple screen reading programs available, including Job Access With Speech (“JAWS”) from Freedom Scientific, Window-Eyes from GW Micro, Dolphin Supernova by Dolphin Computer Access, and open-source products such as NonVisual Desktop Access (“NVDA”).

When using web-based services, such as a social media platform, a computer user typically accesses these services through a web browser, such as Microsoft's Edge or Internet Explorer, Google's Chrome, Apple's Safari, or Mozilla's Firefox. This process is known in the art and generally involves having a web-based service send content over a network to the browser, which displays the content. The browser then sends back text, clicks, and other user-provided content, allowing the user to interact with the service.

To allow sight-impaired users access to web-based services, many screen-reading programs work with one or more web browsers to read the content sent by the web service. Similarly, these programs (along with other assistive-technology programs such as voice-recognition software like Nuance Communication's Dragon Naturally Speaking) work with or through the browser to send text and clicks back to the web-based service based on verbal instructions by the user. For example, a user may say “click on <some link>,” and the program will “click” on that link.

While assistive-technology programs, such as screen readers, have become more advanced and are capable of parsing complex web pages, because these programs are often incapable of determining which parts of a web page should be read first (and which parts should be skipped entirely), the screen-reading experience is often less efficient and less engaging than a traditional screen-viewing experience. This creates difficulties for web-based systems because the services wish to create even more complex and visually-appealing web pages to attract and retain users, but do not want to prevent assistive-technology programs from being able to use them.

SUMMARY

Accordingly, there is a need for a method for a web-based service to determine if a user is using a screen-reading assistive-technology program so that the web service can provide screen-reading users with web pages that are easier for their screen-reading software to parse. From the perspective of a web service, a screen reader is generally passive, in that a web server cannot determine whether content it sent to a client computer is being displayed on a screen or read by a screen reader. However, users of screen readers to audibly read content on screens generally use the assistive-technology to interact with the web services. Accordingly, there is a need to identify assistive-technology programs used to send data (text, clicks, etc.) to a web service. Further, there is a need to determine the type(s) of assistive-technology program being used by the user.

In accordance with some embodiments, a method is performed at a client device with one or more processors and memory storing instructions for execution by the one or more processors. The method includes identifying a trigger event transmitted from a browser on a client device. The method further comprises identifying an inconsistency in the trigger event, wherein the inconsistency correlates with use of an assistive-technology program in conjunction with the browser on the client device. The method further comprises, in response to identifying the inconsistency, flagging a user of the client device as a user of the assistive-technology program. The method comprises, in response to flagging the user of the client device as a user of the assistive-technology program, providing content that is compatible with the assistive-technology program to the client device.

In accordance with some implementations, an electronic system includes one or more processors and memory storing one or more programs configured to be executed by the one or more processors. In some embodiments, a client will interact with the electronic system. The one or more programs include instructions for performing the operations of the method described above. In accordance with some implementations, a non-transitory computer-readable storage medium has stored therein instructions that, when executed by the electronic system, cause the presentation device to perform the operations of the method described above.

Thus, users of screen-reading programs are provided with content that is compatible with their screen-reading programs.

DETAILED DESCRIPTION

It will also be understood that, although the terms first, second, etc. are, in some instances, used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another.

As used herein, the term “exemplary” is used in the sense of “serving as an example, instance, or illustration” and not in the sense of “representing the best of its kind.”

FIG. 1is a block diagram illustrating an exemplary network architecture100in accordance with some implementations. The network architecture100includes one or more clients104-1. . .104-n(where n is an integer greater than or equal to one) and one or more server systems102. One or more networks112communicably connect each component of the network architecture100with other components of the network architecture100. In some implementations, the one or more networks112include public communication networks, private communication networks, or a combination of both public and private communication networks. For example, the one or more networks112can be any network (or combination of networks) such as the Internet, other wide area networks (WAN), local area networks (LAN), virtual private networks (VPN), metropolitan area networks (MAN), peer-to-peer networks, and/or ad-hoc connections.

A client104(e.g.104-1,104-2, . . .104-n) is associated with one or more users106(e.g.,106-1,106-2, . . .106-n). In some implementations, a client104is a personal computer, a mobile electronic device, a wearable computing device, a laptop, a tablet computer, a mobile phone, a feature phone, a smart phone, a digital media player, or any other device capable of capturing and/or transmitting data. In some implementations, clients104include input devices208for receiving user inputs (e.g., keyboards or mice for receiving a first version and a second version of an API and/or an application name, which the clients may store and/or transmit to other components of the network architecture100, such as the server system102). Clients104may be the same type of device (e.g., all desktop personal computers), or may comprise different types of devices.

Users106employ clients104to execute functions, such as functions related to social media systems. In some embodiments, functions may be executed entirely on the clients104. In some such embodiments, the client104may not be connected to a network112or may be disconnected from the network112temporarily.

In some embodiments, the client104will send a function request to a server system102, which will perform the requested function, and the client will display the result. For example, the request may ask the server to return a message. The clients104may access the server systems102through various means via network112. For example, in some embodiments, a client104executes a web browser application that can be used to access a social media service hosted by the server system102. For another example, in some embodiments, the client104may execute a software application that is specific to the social media service (e.g., an “app” running on a smart phone, tablet, or other device).

In some embodiments, the server system102stores and provides content (via the network(s)112) to the users106via the client104. Content stored and served by the server system102, in some implementations, includes lists of functions, user data, and content necessary to perform the functions.

The description of the server system102as a “server” is intended as a functional description of the devices, systems, processors, and/or other components that provide the functionality attributed to the server system102. It will be understood that the server system102may be a single server computer or multiple server computers. The server system102may be coupled to other servers and/or server systems, or other devices, such as other user devices, databases, content delivery networks (e.g., peer-to-peer networks), network caches, and the like. In some embodiments, the server102may communicate with other, unaffiliated, servers. In some implementations, the server system102is implemented by multiple computing devices working together to perform the actions of a server system (e.g., cloud computing). Furthermore, any functionalities performed by the server system102may also be performed additionally, alternatively, and/or concurrently by the one or more clients104. For example, some keyword functions may be fully executable on a client device104without the need to communicate with a server102.

Clients104and the server system102may be communicably coupled in a variety of ways. In some implementations, multiple clients104send requests to perform various functions to a server system102via a network112. In response to receiving the requests, the server system102may return the requested information to the client104.

FIG. 2is a block diagram illustrating an exemplary client200(e.g.,104-1. . .104-n,FIG. 1) in accordance with some implementations. The client200typically includes one or more central processing units (CPU(s), e.g., processors or cores)202, one or more network (or other communications) interfaces210, memory212, and one or more communication buses214for interconnecting these components. The communication buses214optionally include circuitry (sometimes called a chipset) that interconnects and controls communications between system components.

As also shown inFIG. 2, the client200typically includes a user interface204, including output device(s)206and input device(s)208. In some embodiments, output devices206comprise a screen240, speaker241, or screen-reader243. In some embodiments, the speakers241may be headphones or earbuds external to the client200. Such headphones or earbuds may connect to client200via a cord or may communicate with the client200through a wireless protocol or network via a network interface210. In other embodiments, speakers241may comprise internal “built-in” speakers in the client200. In some embodiments, client200may have multiple speakers241. Screen reader243may be a physical device or may be a software application. In some embodiments, screen reader243may comprise both a physical device and a software application. In some embodiments, screen reader243will generate output through speaker241. In other embodiments, screen reader243may have its own output mechanism, such as its own speaker.

In some implementations, the input devices208include one or more of a keyboard242, mouse244, trackpad245, camera246, microphone248, or assisted input application249. Alternatively, or in addition, the user interface240may include a display device that may include a touch-sensitive surface, in which case the screen240may also comprise a touch-sensitive display (e.g., a “touchscreen”). In user devices that have a touch-sensitive display, a physical keyboard is optional (e.g., a soft keyboard may be displayed when keyboard entry is needed). Furthermore, some user devices102may use a microphone248and an assisted input application249to supplement or replace the keyboard242. In some embodiments, the microphone248may be a “built-in” condenser microphone located inside the client200. In some embodiments, microphone248may comprise an external microphone, connected to client200via a cord or through a wireless protocol or network via a network interface210. In some embodiments, client200will contain multiple microphones248.

The assisted input application249may comprise voice recognition software to receive audible input via a microphone248. The assisted input application249may transcribe the received sounds (such as a spoken language) into text which is then input to client200in a similar manner to text input through a keyboard242. The assisted input application249may also accept commands to perform actions commonly performed by a mouse244or trackpad245, such as moving a cursor or “clicking” on an object (or text). For example, an assisted input application249may receive a verbal command to “move cursor to <somewhere>” or to “click on <some link>” and will take the appropriate action(s), such as moving the cursor or “clicking” on the link. It should be noted that assistive input application249may process commands such as “click” commands differently than mice244or trackpads245. For example, while, generally, a mouse244can click only where a cursor is located, an assistive input application249may be capable of clicking elsewhere on or off of the display. For example, if a user106(FIG. 1) issued a command to “click on <some image>,” assistive input application249may “click” on the specified image without moving the cursor to that image. Assisted input application249may be a physical device or may be a software application. In some embodiments, assisted input application249may comprise both a physical device and a software application. In some embodiments, assisted input application249will receive input from microphone248. In other embodiments, assisted input application249may have its own input mechanism, such as its own microphone.

In some implementations, the one or more network interfaces210include wireless and/or wired interfaces for receiving data from and/or transmitting data to a server system102and/or other devices or systems. In some implementations, data communications are carried out using any of a variety of custom or standard wireless protocols (e.g., NFC, RFID, IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth, ISA100.11a, WirelessHART, MiWi, etc.). Furthermore, in some implementations, data communications are carried out using any of a variety of custom or standard wired protocols (e.g., USB, Firewire, Ethernet, etc.). For example, in some implementations, the one or more network interfaces210includes a wireless LAN (WLAN) interface for enabling data communications with the server system104(via the one or more network(s)112,FIG. 1).

Memory212includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices; and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory212may optionally include one or more storage devices remotely located from the CPU(s)202. Memory212, or alternately, the non-volatile memory solid-state storage devices within memory212, includes a non-transitory computer-readable storage medium. In some implementations, memory212or the non-transitory computer-readable storage medium of memory212stores the following programs, modules, and data structures, or a subset or superset thereof:an operating system216that includes procedures for handling various basic system services and for performing hardware dependent tasks;a browser type detection module270for determining the type of browser224used on the client200;network communication module(s)218for connecting the client104to other computing devices (e.g., server system102) via the one or more network interface(s)210(wired or wireless);a user interface module220that receives commands and/or inputs from a user106via the user interface204(e.g., from input devices208), and provides outputs for display by the user interface204(e.g., the output devices206); andone or more client application modules222, including the following modules (or sets of instructions), or a subset or superset thereof:a web browser module224(e.g., Internet Explorer or Edge by Microsoft, Firefox by Mozilla, Safari by Apple, or Chrome by Google) for accessing, viewing, and interacting with web sites (e.g., a social-networking web site provided by the server system102),a social media application module226for allowing a user106to access a social media system;a voice recognition module228, to receive and parse spoken language;assisted technology module(s)232for performing tasks related to providing assisted access to computer input and output functions, including but not limited to:a screen reading module234for converting content on a screen240into audible/spoken word;a text entry module236for accepting verbal entry of text; anda link selection module238for accepting verbal commands to select or “click” on a link, such as links from browser module224or social media application module226;a cursor location detection module229for determining the location of a cursor on screen240;a mouse event detection module225for determining the occurrence of mouse events, including, but not limited to events generated by mouse244, trackpad245, or assisted input application249;a text entry detection module227for detecting the entry of text, including, but not limited to, events generated by keyboard242, assisted input application249, text entry module236, or voice recognition module228.one or more client data module(s)250for handling the storage of and access to content, including but not limited to:media storage260, for storing media, including waveform or other audio data used to map words into spoken language;a voice recognition database252, containing data to map waveforms (such as those generated by speech in one or more languages or dialects) or other audio data into text; anda user information database254, for containing information regarding106;other optional client application module(s)230, such as applications for word processing, calendaring, mapping, weather, stocks, time keeping, virtual digital assistant, presenting, number crunching (spreadsheets), drawing, instant messaging, e-mail, telephony, video conferencing, photo management, video management, a digital music player, a digital video player, 2D gaming, 3D gaming, virtual reality, electronic book reader, and/or workout support.

The optional browser type detection module270provides functions for detecting the type of browser224being used on client200. The “type” of browser may include data including the name of the browser224(e.g., Chrome or Firefox), the version of the browser (e.g., 1.1 or 2.0), configuration, or other information. In some embodiments, this data will include the operating system310of the client200. Other browser-related data may also be detected. For example, information indicating the screen resolution or the size and/or fonts of text may be detected. This data may be stored in a client data module250. In some embodiments, the browser type data may be associated with a user106(FIG. 1) and stored in user information database254or in user database332(FIG. 3).

The optional voice recognition module228monitors waveforms, such as those from sound to detect spoken language. In some embodiments, voice recognition module228will convert the spoken language into text (such as text to be input by text entry module236) or commands (such as commands to direct assisted application249). In some embodiments, such commands may include a command to “click” on a link using link selection module238. In some embodiments, voice recognition module228may be a part of assisted input application249. In some embodiments, it may be a part of an assisted technology module232. In some embodiments, it may be a component of a screen reader243, such as JAWS, Dolphin, or NVDA. In some embodiments, it may comprise a stand-alone commercial application, such as Dragon NaturallySpeaking.

Optional cursor location detection module229detects the location of a cursor on a screen, such as screen240. In some embodiments, it will report on an absolute location of a cursor on a screen240(or on a system for a multi-screen computer containing multiple screens240). In some embodiments, it will report a relative screen location such as the location of a cursor on a secondary (or tertiary) screen240or on a window on screen240. In some embodiments, cursor location module229will report whether the cursor is located on an object (such as an image) on a screen or window. In some embodiments, cursor location detection module229(or another client application module222) will send these events to a cursor detection module318(FIG. 3) on a server300.

Optional assisted technology module232provides features useful to a user106with a disability such as sight impairment. In some embodiments, optional assisted technology module232includes a screen reading module234that reads text displayed on a screen240. In some embodiments it may also read meta-data related to the text, such as data describing the font, size, or style of the text or its location on the screen or a window. In some embodiments, screen reading module234may be a component of a commercial (or open-source) disability program such as JAWS, Dolphin, or NVDA. In some embodiments, it may be a function of or be associated with a physical device, such as a screen reader243. In some embodiments, optional assisted technology module232includes a text entry module236. Text entry module236allows for the entry of text from an assisted input application249. In some embodiments, text entry module236may be a part of a physical assisted input application249device. In some embodiments, text entry module236may comprise (or be a component of) a voice recognition module228, such as Dragon NaturallySpeaking. In some embodiments, assisted technology module232may comprise a link selection module238, which allows the selection of a “link” on a web browser module224or social media application module226.

The optional mouse event detection module225detects mouse events. These events may be generated by mice244, trackpads245, or assisted input applications249. These events may include, but are not limited to, mouse-up, mouse-down, mouse-click, light press (tap), and strong press. In some embodiments, mouse event detection module225will also detect events or meta-data related to mouse events, such as indications that an event is (or is not) reliable. In some embodiments, mouse event detection module225(or another client application module222) will send these events to a mouse event detection module320(FIG. 3) on a server300.

The optional text entry detection module227detects the entry of text by a user106. In some embodiments, text entry may be from a keyboard242(physical or virtual) or an assisted input application249. In some embodiments, text entry detection module227will detect text from the text entry module236component of the assisted technology module232.

Voice recognition database252stores and indexes waveform and/or other audio data sufficient to generate text based on spoke language. In some embodiments, the waveform data will include data with respect to various languages such as English, Spanish, or Chinese. In some embodiments, waveform data will include data with respect to language dialects, such as Southern, Northeastern, or Texan for English. In some embodiments, the database may be updated based on waveforms read by voice recognition module228for a specific user106. It may comprise one or more types of databases, such as text, graph, dimensional, flat, hierarchical, network, object-oriented, relational, distributed, and/or XML, databases. In some embodiments, voice recognition database252may be accessed publically. In other embodiments, voice recognition database252may be accessed only by assisted input application249, voice recognition module228, one or more assisted technology modules232, social media application module226and/or other client application modules230. In some embodiments, this database may be encrypted.

Optional user information database254contains information related to users106of clients200. In some embodiments, this data will comprise demographic data regarding the users106. In some embodiments, it may comprise information about what types of browser modules224are used by the users106. In some embodiments, this data may be updated by browser type detection module270. It may comprise one or more types of databases, such as text, graph, dimensional, flat, hierarchical, network, object-oriented, relational, distributed, and/or XML databases. In some embodiments, user information database254may be accessed publically. In other embodiments, user information database254may be accessed only by assisted input application249, one or more assisted technology modules232, social media application module226and/or other client application modules230. In some embodiments, this database may be encrypted.

AlthoughFIG. 2illustrates client200in accordance with some implementations,FIG. 2is intended more as a functional description of the various features that may be present in one or more clients200than as a structural schematic of the implementations described herein. In practice, items shown separately could be combined and some items could be separated. For example, some items shown separately inFIG. 2could be implemented in single applications and single items could be implemented by one or more applications. The actual number of applications used to implement the client200, and how features are allocated among them, will vary from one implementation to another.

Similarly, some hardware items shown separately could be combined and some items could be split into multiple hardware items. For example, assisted input application249may contain its own sound sampling device, or may use microphone248. For another example, screen reader243, if implemented as a physical device, may have its own microphone and/or sound generating equipment, or may use one or more microphones248and/or speakers241of client200. Further, some components could be implemented as hardware, as software, or as a combination of hardware and software. For example, assisted input application249may be a physical device or may be a software application. When assisted input application249is a physical device, it may work in conjunction with voice recognition module228. In some embodiments, voice recognition module228may be a component of assisted input application249. For simplicity, in describing the voice entry function in method400(FIG. 4), infra, the voice command and voice-based text entry technology will be described as being implemented by assisted input application249, but one of skill in the art will appreciate that the invention equally covers pure software and device-software hybrid types of voice command technology. For another example, screen reader243may be implemented as a physical hardware device or as a software application, such as screen reading module234, or as a combination of hardware and software. For simplicity, in describing the screen reading function in method400, infra, the screen reading technology will be described as being implemented by physical screen reader243, but one of skill in the art will appreciate that the invention equally covers pure software and device-software hybrid types of screen reading technology.

FIG. 3is a block diagram illustrating an exemplary server system300(e.g., server102-1,102-n,FIG. 1) in accordance with some implementations. The server system300typically includes one or more central processing units/cores (CPUs)302, one or more network interfaces304, memory306, and one or more communication buses308for interconnecting these components.

Memory306includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices; and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory306, optionally, includes one or more storage devices remotely located from one or more CPUs302. Memory306, or, alternatively, the non-volatile solid-state memory device(s) within memory306, includes a non-transitory computer-readable storage medium. In some implementations, memory306, or the non-transitory computer-readable storage medium of memory306, stores the following programs, modules, and data structures, or a subset or superset thereof:an operating system310that includes procedures for handling various basic system services and for performing hardware dependent tasks;a network communication module312that is used for connecting the server system300to other computing devices (e.g., clients200(FIG. 2) or other server systems300) via one or more network interfaces304(wired or wireless) connected to one or more networks112(FIG. 1) such as the Internet, other WANs, LANs, PANs, MANs, VPNs, peer-to-peer networks, content delivery networks, ad-hoc connections, and so on;one or more server application modules314for enabling the server system300to perform various functions, the server application modules314including, but not limited to, one or more of:an assistive technology detection module316for providing assistive technology services to users106of clients200;a cursor detection module318, for detecting the location of a cursor on or off of a screen240(FIG. 2) of a client200;a mouse event detection module320, for detecting a mouse event from a client200;a text entrance detection module322, for detecting the entry of text on a client200;a link click detection module234, for detecting that a client200has clicked on a link sent to the client, such as by web server module328; anda browser detection module326, for detecting the type of browser module224(FIG. 2) used by client200;an optional web server module328, for serving web pages to a client200or another server300;one or more server data module(s)330for handling the storage of and access to content, including but not limited to:a user database332, for storing user data;an inconsistency database334, for storing inconsistencies that may be used by assistive technology detection module316to determine that a user106of a client200is using an assistive-technology program;media store338, for storing media data, including, but not limited to media served by web server module328;web page storage336, for storing web pages, including, but not limited to pages served by web server module328;

The optional cursor detection module318detects the location of a cursor on a screen240(FIG. 2) of a client200. In some embodiments, cursor detection module318will work in conjunction with cursor location detection module229(FIG. 2). In some embodiments, the location of the cursor will be sent to cursor detection module318by cursor location detection module229.

The optional mouse event detection module320detects “mouse” events of a client200(FIG. 2). These events may be generated by mice244, trackpads245, or assisted input applications249of clients200. These events may include, but are not limited to, mouse-up, mouse-down, mouse-click, light press (tap), and strong press. In some embodiments, mouse event detection module225will also detect events or meta-data related to mouse events, such as indications that an event is (or is not) reliable. In some embodiments, mouse event detection module320will work in conjunction with mouse event detection module225(FIG. 2). In some embodiments, the event will be sent to mouse event detection module320by mouse event detection module225.

The optional link click detection module324detects links clicked by a user106(FIG. 1) of client200(FIG. 2) on web pages. In some embodiments, it will also detect selections of other UI elements in a browser224, such as radio boxes, checkboxes, text boxes, buttons, etc. In some embodiments, click detection module324may detect clicks on UI elements on a screen240. In some embodiments, the web pages will be served to the client200by web server module328and web server module328will report the click to link click detection module324. In some embodiments, link click detection module324will work in conjunction with link selection module238(FIG. 2). In some embodiments, the indication of the link being clicked will be sent to link click detection module324by link selection module238.

The optional browser detection module326detects the type of browser used by a user106(FIG. 2) on client200(FIG. 2). In some embodiments, the browser module224will report its type to web server module328, which will report the type to browser detection module326. In some embodiments, browser detection module326will work in conjunction with browser type detection module270(FIG. 2). In some embodiments, the indication of the link being clicked will be sent to browser detection module326by browser type detection module270. In some embodiments, browser detection module326will determine the browser type based on data received from client200or on meta-data associated with this data, such as timing data.

Text entrance detection module322detects text entered by client200(FIG. 2) and also meta-data associated with that text, such as timing data. Timing data may include speed, cadence/consistency, pauses (or lack thereof), or corrections (or lack thereof). In some embodiments, the text (and/or meta-data) will be sent to text entrance detection module322by text entry module236. In some embodiments, text detection module322is used along with other assistive technology detection module(s)316to determine that a user106(FIG. 1) is using assistive technology, such as a screen reader243.

Inconsistency database334contains information about inconsistencies in user input. In some embodiments, this information allows assistive technology detection module(s)316to determine that a user106uses assistive-technology programs. In some embodiments, this information may be entered or updated by one or more assistive technology detection modules316. It may comprise one or more types of databases, such as text, graph, dimensional, flat, hierarchical, network, object-oriented, relational, distributed, and/or XML databases. In some embodiments, inconsistency database334may be accessed publically. In other embodiments, inconsistency database334may be accessed only by one or more assisted technology detection modules316and/or other server application modules. In some embodiments, this database may be encrypted.

Optional user database332contains information related to users106(FIG. 1). In some embodiments, this data will comprise demographic data regarding the users106. In some embodiments, it may comprise information about what types of browser modules224(FIG. 2) or client operating systems216are used by the users106. In some embodiments, this data may be updated by browser detection module326. It may comprise one or more types of databases, such as text, graph, dimensional, flat, hierarchical, network, object-oriented, relational, distributed, and/or XML databases. In some embodiments, user database332may be accessed publically. In other embodiments, user database332may be accessed only by one or more assisted technology detection modules316, and/or other server application modules. In some embodiments, this database may be encrypted.

In some implementations, the server system300includes web or Hypertext Transfer Protocol (HTTP) servers, File Transfer Protocol (FTP) servers, as well as web pages and applications implemented using Common Gateway Interface (CGI) script, PHP Hypertext Preprocessor (PHP), Active Server Pages (ASP), Hyper Text Markup Language (HTML), Python, Extensible Markup Language (XML), Java, JavaScript, Asynchronous JavaScript and XML (AJAX), XHP, Javelin, Wireless Universal Resource File (WURFL), and the like.

AlthoughFIG. 3illustrates the server system300in accordance with some implementations,FIG. 3is intended more as a functional description of the various features that may be present in one or more server systems than as a structural schematic of the implementations described herein. In practice, items shown separately could be combined and some items could be separated. For example, some items shown separately inFIG. 3could be implemented on single servers and single items could be implemented by one or more servers. The actual number of servers used to implement the server system300, and how features are allocated among them, will vary from one implementation to another and, optionally, depends in part on the amount of data traffic that the server system handles during peak usage periods as well as during average usage periods.

Each of the above identified modules stored in memory212and306corresponds to a set of instructions for performing a function described herein. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or otherwise re-arranged in various implementations. In some implementations, memory212and306optionally store a subset or superset of the respective modules and data structures identified above. Furthermore, memory212and306optionally store additional modules and data structures not described above.

FIG. 4is a flow diagram illustrating a method400of determining if a user is using a screen-reading assistive-technology program.

The steps of the method400may be performed by any combination of one or more clients104(FIGS. 1 and 2) and/or server system102(FIGS. 1 and 3).FIG. 4corresponds to instructions stored in computer memory (e.g., memory212of the client104,FIG. 2; memory306of the server system102,FIG. 3; or other computer-readable storage medium. In some embodiments, method500may be performed entirely on a client104.

In some embodiments, in performing the method400, in step402, an electronic system having one or more processors and memory storing instructions for execution by the one or more processors provides an imperceptible link to a client device. For example, the system may provide a link that does not render on a screen240(FIG. 2), but which would be read by a screen reader243or another assistive-technology program. In some embodiments, the electronic system is a server300(FIG. 3).

In some embodiments, the electronic system will identify (404) a trigger event transmitted from a web browser on the client device200(FIG. 2). In some embodiments, rather than a browser module224, the user106may be using a more specific client application module for a web-based service, such as a social media application module226. For simplicity, generally only a web browser will be described, but one of skill in the art will appreciate that the invention applies equally well to custom client modules for a web service and client applications.

There are multiple methods of identifying a trigger event. For example, in some embodiments, the method can use “plug-in” client applications that work with web browser modules224to detect events. Based on the type of event being detected, these methods may include mouse event detection modules225, a text entry detection module227, or a cursor location detection module225. In some embodiments, multiple such modules (and potentially other client application modules230) will be used together. For example, in some embodiments, the method can use code transmitted from a server300to a client200that executes on the client200in a browser module224. Examples include JavaScript, ActiveX, and other such languages. In some embodiments, both a “plug-in” and server-delivered script code will be used. In some embodiments, other techniques will be used, either by themselves, or in conjunction with a plug-in module or server-delivered script code.

In some embodiments, the trigger event is a text entry event. For example, text entry detection module227(FIG. 2) may notify text entrance detection module322(FIG. 3) on server300that text has been entered on a browser module224.

In some embodiments, the trigger event is a click event (406). In some embodiments, the trigger event is (408) a click event on the imperceptible link. In some embodiments, the click event is a click on an imperceptible link on a page served by web server module328. In some embodiments, a service on server300will generate a page containing or more imperceptible links to a user106that has already been identified as being a likely user of assistive technology, such as a screen reader243.

Sometimes, a source of the trigger event is (410) a force-sensitive input device. In some embodiments, the force-sensitive input device is a touchpad (412), such as trackpad245. The force-sensitive trackpad245may be capacitive. For example, in some embodiments, trackpad245may be Synaptics' ForcePad. For another example, the force-sensitive input device may be an Apple MacBook Force Touch Trackpad. The force-sensitive input device may also be a mouse244with a pressure-sensitive button or grip.

In some embodiments, the electronic system will obtain (416) an identifier or a name for the browser224(FIG. 2) on the client200. In some embodiments, the identifier will be the name. In some embodiments, the identifier may comprise a code or string that contains information about the browser224, including one or more of: name, version, configuration, computer type, or operating system. For example, the identifier may indicate the type of browser224is Google's Chrome v54.0.2840.99 m on a Windows 10 64-bit operating system216running on a computer comprising an Intel Core i5 vPro CPU202. In some embodiments, the “browser” may be a social media application module226. In such embodiments, the identifier may also comprise information related to the social media platform associated with the social media application module226. In some embodiments, the identifier may comprise an XML document.

In some embodiments, the electronic system will obtain (418) an identifier or name for an operating system216(FIG. 2) for the client200. In some embodiments, the identifier will be the name of the operating system. For example, the identifier could be “Windows 10” or “MacOS Leopard.” In some embodiments, the identifier may comprise a code or string which contains information about the operating system216, including one or more of: name, version, configuration, or computer type. For example, the identifier may indicate that the operating system216is Windows 10 v1.0.1 running on a Dell Inspiron computer running an Intel Core i7 CPU202. In some embodiments, the identifier may comprise an XML document.

In some embodiments, the electronic system will identify (420) an inconsistency in the trigger event, where the inconsistency correlates with the use of an assistive-technology program in conjunction with the browser224(FIG. 2) on the client200. In some embodiments, the assistive-technology program comprises a screen reader243. In some embodiments, the assistive-technology program comprises an assisted input application249. In some embodiments, the assistive-technology program comprises a voice recognition module228. It will be appreciated that not all inconsistencies will by identified by every combination of client operating system216type, browser module224type, or assistive technology program type. It will also be appreciated that the use of some combinations of client operating system216type, browser module224type, or assistive technology program type may only result in one identified inconsistency and others result in identification of multiple inconsistencies.

In some embodiments, a list of inconsistencies will be stored in inconsistency database334. Those of skill in the art will appreciate that while certain inconsistencies are described herein, additional inconsistencies will also indicate the use of assistive-technology programs such as screen readers243(FIG. 2) by a user106(FIG. 1) and that detecting inconsistencies such as these is within the scope of this invention.

In some embodiments, the inconsistency is determining that the text event is text entered in a manner inconsistent with text being entered on a keyboard242(FIG. 2). For example, text entrance detection module322(FIG. 3) may determine that the text was entered with a cadence that indicates it is being generated by an assisted input application249.

In some embodiments, the inconsistency is determining (422) that the click event is on the imperceptible link. For example, a link that does not render on a screen240(FIG. 2), but which is read by a screen reader243may be clicked on by a user106of a client200, indicating that the user106is using a screen reader. This is an example of an active identification, as it requires that web server module328(FIG. 3) on server300serve the client200a webpage that contains the imperceptible link.

Often, it is advantageous to detect that a user106is using an assistive-technology program, such as a screen reader243(FIG. 2) passively, without providing the client200with any special or modified web pages. This is helpful, for example, when the service does not have control over the specific web pages being served or does not want to send modified web pages to a client200. As discussed previously with respect to steps404and406, in some embodiments, trigger events, including click events, are received from clients200. One such passive technique is to identify “inconsistencies” in those click events and, based on those inconsistencies, determine that a user106associated with the client device likely uses assistive-technology programs.

In some embodiments, the inconsistency is receiving (424) the click event without also receiving a preceding mouse-down event. For example, because a physical mouse244(FIG. 2) would generate a mouse-down event when the “clicking” button was depressed prior to the “click,” the electronic system can determine that the click event may not have originated from a physical mouse244and may have originated from an assistive-technology program instead. Similarly, in some embodiments, the inconsistency is receiving (426) the click event without also receiving a following mouse-up event. For example, because a physical mouse244(FIG. 2) would generate a mouse-up event when the “clicking” button was released after a click, the electronic system can determine that the click event may not have originated from a physical mouse244and may have originated from an assistive-technology program instead. In some embodiments, certain versions of the Microsoft Edge browser module224are known not to generate mouse-up events after click events when used with an assisted input application249as an input device208.

In some embodiments, the inconsistency is receiving (428) the click event on an object when the cursor location for the click event is outside the object. For example, a click event indicating that an image on a screen has been selected may be received when the cursor on client200is not on the image. For example, the cursor location may be at a valid positive location on a screen240, but is still outside the object being clicked on. Because a typical click from a mouse244or trackpad245requires that the cursor be located on the object (e.g., an image or text link), the electronic system can determine that the click may have originated from an assistive-technology program. Additionally, some reported cursor locations may further indicate an assistive-technology program such as a screen reader243is being used. For example, the transmitted cursor location may comprise a negative value (430), which is an impossible cursor location. For another example, the cursor location may comprise a position of zero in a first direction, an offset of zero in the first direction, a position of zero in a second direction, and an offset of zero in the second direction (432). While such a cursor location is not impossible, it may be indicative that the cursor is not being used as a means to select objects to click. In some embodiments, certain versions of Microsoft's Edge, Mozilla's Firefox, and Google's Chrome browser modules224are known to report such cursor locations when used with an assisted input application249as an input device208.

In some embodiments, input sources associated with a click event and corresponding to input devices208(FIG. 2) are received. For example, the received input source may indicate that the input device208associated with a click event is a mouse244, a keyboard242, or a trackpad245. For example, in some embodiments, certain versions of Mozilla's Firefox browser module224report an enumerated list of input sources from 1-7 for physical devices. In some embodiments, the identified inconsistency comprises receiving (434) an impossible input source associated with the click event. For example, the input source may be an impossible value such as zero or a negative number. In some embodiments, the input source may be null. In some embodiments, certain versions of Mozilla's Firefox browser module224report an input source of zero when used with an assisted input application249as an input source208.

In some embodiments, it may be possible to receive an “impossible” input source, such as when a “click event” is generated by a script running in the browser module224itself. For example, many popular websites include JavaScript or ActiveX code that runs on a browser224, such as Chrome, Firefox, or Internet Explorer, and causes the browser224to return click events, despite no action being taken by a user106. For example, in some embodiments, Mozilla's Firefox browser module224will send a click event with an input of zero in response to a “click” resulting from scripts in web pages. As explained above with respect to step434, zero is the same input source certain versions of Mozilla's Firefox report when being used with assistive-technology programs. Accordingly, to decrease the likelihood of mistakenly determining that a user106is using an assistive-technology program due to script code running on the client browser, in some embodiments, the electronic system will receive a property associated with the click event that indicates the event is reliable (436). For example, the property may be “isTrusted.”

In some embodiments, the click event is generated by a force-sensitive input device208(FIG. 2), such as a force-sensitive mouse244or a force-sensitive trackpad245. Such devices may report a force level associated with click events, indicating the amount of force applied to the click, allowing some client application modules222to respond differently based on the force of the “click.” For example, in certain versions of Apple's Safari browser224later than v7, Safari will send a force level to a web server module328when a click is performed on a webpage served by that web server module328. In some embodiments, the identified inconsistency is receiving (438) an impossible force level associated with the trigger event. For example, the force level may be zero (440). In some embodiments, Apple's Safari browser will report a force level of zero when it is used with an assisted input application249as an input device208. For another example, the force level may be null. For another example, the force level may be negative. For another example, the force level may be positive, but may be outside the range permitted by the specific input device208.

In some embodiments, it may be possible to receive an “impossible” force level, such as when a “click event” is generated by a script running in the browser module224itself. For example, many popular websites include JavaScript or ActiveX code that runs on a browser224, such as Chrome or Internet Explorer, and may cause the browser224to report click events, despite no action being taken by a user106. To decrease the likelihood of mistakenly determining that a user106is using an assistive-technology program due to script code running on the client browser, in some embodiments, the electronic system will receive a property associated with the click event that indicates the event is reliable (442). For example, the property may be “isTrusted.”

In some embodiments, the electronic system will verify (444) that the name of the browser module224(FIG. 2) corresponds to the inconsistency. For example, some inconsistencies apply to certain browser modules. For example, a given inconsistency may apply to Edge and Internet Explorer, but may not apply to Chrome, Firefox, or Safari. For example, certain versions of Microsoft's Edge browser module224on the Windows operating system216trigger a mouse-down event before a click event when used with assistive-technology programs, but do not trigger a mouse-up event. Certain older versions of Apple's Safari browser module224report mouse-up events when used with assistive-technology programs, but do so outside the boundaries of the object being clicked. Similarly, certain versions of Mozilla's Firefox, Google's Chrome, and Microsoft's Edge all report click events at the top corner of a window when used with assistive-technology programs, but Microsoft's Internet Explorer reports these events as coming from off of the screen240. Further, it is possible that a perceived inconsistency could occur when a user106was not using assistive-technology programs. For example, an event could have not been detected by mouse event detection module320(FIG. 3) or an error on client200resulted in the event not being generated or not being properly transmitted. For this reason, it is important to identify not only an inconsistency, but also to identify the browser type being used when the inconsistency was detected. By doing so, the electronic system can have a higher level of confidence in its determination that the detected inconsistency was the result of use of an assisted input application249.

In some embodiments, in response to identifying the inconsistency, the method flags (446) a user of the client device as a user of the assistive-technology program. For example, the electronic system may identify a user106(FIG. 1) as using a particular assistive-technology program or an assistive-technology program in general. In some embodiments, the flagging is performed (448) in response to verifying that the name of the browser corresponds to the inconsistency. This helps ensure that users are not improperly flagged as using assistive-technology programs. When a user106is flagged as being a user of an assistive-technology program, the user's information in user information database254(FIG. 2) or user database332(FIG. 3) may be updated. In some embodiments, the fact that the user uses an assistive technology program will be stored. In some embodiments, where it is possible to identify one or more specific assistive-technology programs used by the user (e.g., JAWS and Dolphin for a specific user106), the names or identifiers of the specific programs may also be stored and associated with the user.

In some embodiments, the method adds (450) a first entry to a log indicating that the assistive-technology program is being executed on the client device200. The log entry is associated with the user106(FIG. 1) of the client device.

In some embodiments, the method determines (452), based on the log entry, that the assistive-technology program has been used on the client device200. It will be appreciated that the log entry may be a log entry generated by the method (such as in step450). However, in some embodiments, the log entry may be from a log obtained from an external source. For example, the log could be purchased from an information broker or “big data” company. In some embodiments, log entries may be obtained from an external source, but verified by testing for one of the inconsistencies (or vice-versa).

In some embodiments, in response to flagging the user of the client device as a user of an assistive-technology program, the method provides (454) content that is compatible with the assistive-technology program to the client device200. For example, if a user106(FIG. 1) was identified as using a screen reader243program, such as JAWS or NVDA, content on a web page will be delivered that is designed to be compatible with the screen reading program. By providing web pages that are compatible with the user's assistive-technology program(s), services on the electronic system will be better able to engage users using such technologies. In some embodiments, this will be accomplished by having separate versions of web pages (or separate versions of portions of web pages) that are served by web server module328. For example, when the assistive technology detection module316determines that a user106is using an assistive-technology program, it may instruct web server module328to pull and serve a different web page (or a different portion of a web page) from web page storage336than web server module328would have otherwise pulled and served.

For example, a web service which had web pages containing, e.g., tables, columned content, hover-sensitive content (such as content which only appears when a cursor is on a screen location for a predetermined amount of time), pop-up or “scroll-up” windows, or other content that is not properly “read” by screen-reading programs such as screen reader243, can create separate web pages which do not contain this content, or which contain the content formatted in a manner making it “readable” by the screen-reading programs. Additionally, even when web pages are “readable” by screen-reading programs, some formats of web pages may provide a higher reading experience for a user106. Accordingly, web-based services can tailor certain versions of its pages to those users. By having separate versions of web pages, a service will be able to make enhancements to its web pages to improve the experience of users106who interact with the service via a screen240, while ensuring that the enhancements do not degrade the experience of users of assistive-technology programs.

In some embodiments, providing the content to the client device is performed (456) in accordance with determining, based on the log entry that the assistive-technology program has been used on the client device. This can be used either as a primary means of deciding to provide assistive-technology compatible content to the client device200, or as a means of verifying that a determination made based on an identified inconsistency (or multiple identified inconsistences) was correct. As discussed with respect to step452, the log entry may have been generated by the method (e.g., step450) or it may have been obtained from an external source.

In some embodiments, the method assigns (458) the user to a cohort of users of assistive-technology programs. This cohort may comprise users of multiple similar assistive-technology programs, such as screen readers, or it may comprise only users of a single assistive-technology program, such as JAWS. The association of the user106to the cohort may be stored in, e.g, user database332(FIG. 3). In some embodiments, the method obtains (460) identifiers for browsers for the users in the cohort and updates a database with the identifiers. For example, if a specific user was flagged as using an assistive-technology program, such as a screen reader243, on Chrome, that information may be stored in user information database254or user database332. By keeping track of the browsers used by users of assistive-technology programs, a service can improve the accuracy of its determinations that a user likely uses an assistive-technology program. Further, such data may be beneficial in tailoring web pages associated with the service to work on assistive-technology programs on specific browser types.

In some embodiments, the method provides (461) all users in the cohort with content compatible with assistive-technology programs. This process is described previously with respect to step454.

In some embodiments, the method updates (462) a database with the identifiers for the browser224(FIG. 2) and the operating system216used by the user106. In some embodiments, the method will update user information database254, which will permit code (such as JavaScript) executing on a browser module224to determine that assistive-technology module232is being used, allowing the code to report that data to other web server modules328(FIG. 3) associated with other services on servers300. In some embodiments, the method will update user database332with the browser224and the operating system216used by the user. This permits a service associated with server300to send assistive-technology compatible web pages to the user106, even when the user is on a different client200or is using a different browser module224or social media application module226. This data can also be used for detecting trends of browser usage in the sight-impaired community, such that when browser type detection module270subsequently detects the user106using a different browser module224than the one stored in the database(s), the record can be noted of the change. As similar changes are recorded for more users in a cohort, probability analysis can be done to identify trends, allowing a service, such as a social media service, to redesign pages in anticipation of more users in a cohort moving to the newer browser type. Further, as discussed with respect to step444, certain inconsistencies apply only to a subset of browser types. However, newer browser modules224are being developed and newer versions of existing ones are being released. As described earlier, inconsistencies can vary even within different versions of the same browser224. By keeping track of the browsers being used by users106, especially those used by users determined to be using assistive-technology programs, a service can associate detected inconsistencies with newer browsers or newer versions of existing browser.

Although some of various drawings illustrate a number of logical stages in a particular order, stages which are not order dependent may be reordered and other stages may be combined or broken out. Furthermore, in some implementations, some stages may be performed in parallel and/or simultaneously with other stages. While some reordering or other groupings are specifically mentioned, others will be apparent to those of ordinary skill in the art, so the ordering and groupings presented herein are not an exhaustive list of alternatives. Moreover, it should be recognized that the stages could be implemented in hardware, firmware, software, or any combination thereof.